Nov 03 00:04:33.312402 ip-10-0-133-161 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 00:04:33.714874 ip-10-0-133-161 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 00:04:33.714874 ip-10-0-133-161 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 00:04:33.714874 ip-10-0-133-161 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 00:04:33.714874 ip-10-0-133-161 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 00:04:33.718486 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718393    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:04:33.718486 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718480    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:04:33.718486 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718485    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:04:33.718486 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718490    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:04:33.718486 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718495    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:04:33.718782 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718500    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:04:33.718782 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718504    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:04:33.718782 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718507    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:04:33.718782 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718512    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:04:33.718782 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718516    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:04:33.718782 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718519    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:04:33.718782 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718524    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:04:33.718782 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718528    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:04:33.718782 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718532    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:04:33.718782 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718536    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:04:33.718782 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718540    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:04:33.718782 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718544    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:04:33.718782 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718548    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:04:33.718782 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718552    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:04:33.718782 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718556    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:04:33.718782 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718560    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:04:33.718782 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718564    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:04:33.718782 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718578    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:04:33.718782 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718582    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:04:33.718782 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718586    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:04:33.719595 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718589    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:04:33.719595 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718593    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:04:33.719595 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718598    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:04:33.719595 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718601    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:04:33.719595 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718606    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:04:33.719595 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718610    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:04:33.719595 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718614    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:04:33.719595 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718618    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:04:33.719595 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718622    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:04:33.719595 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718626    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:04:33.719595 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718631    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:04:33.719595 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718635    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:04:33.719595 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718639    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:04:33.719595 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718644    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:04:33.719595 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718648    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:04:33.719595 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718653    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:04:33.719595 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718657    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:04:33.719595 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718661    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:04:33.719595 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718665    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:04:33.720095 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718670    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:04:33.720095 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718674    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:04:33.720095 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718678    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:04:33.720095 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718683    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:04:33.720095 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718687    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:04:33.720095 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718691    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:04:33.720095 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718696    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:04:33.720095 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718700    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:04:33.720095 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718704    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:04:33.720095 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718709    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:04:33.720095 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718717    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:04:33.720095 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718723    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:04:33.720095 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718729    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:04:33.720095 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718734    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:04:33.720095 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718739    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:04:33.720095 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718743    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:04:33.720095 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718748    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:04:33.720095 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718752    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:04:33.720095 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718757    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:04:33.720614 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718761    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:04:33.720614 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718766    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:04:33.720614 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718770    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:04:33.720614 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718774    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:04:33.720614 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718779    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:04:33.720614 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718783    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:04:33.720614 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718790    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:04:33.720614 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718797    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:04:33.720614 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718804    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:04:33.720614 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718811    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:04:33.720614 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718817    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:04:33.720614 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718822    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:04:33.720614 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718827    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:04:33.720614 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718831    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:04:33.720614 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718836    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:04:33.720614 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718841    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:04:33.720614 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718845    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:04:33.720614 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718850    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:04:33.720614 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718854    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:04:33.720614 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718858    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:04:33.721583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718863    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:04:33.721583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718867    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:04:33.721583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718871    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:04:33.721583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718876    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:04:33.721583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718881    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:04:33.721583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718885    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:04:33.721583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718889    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:04:33.721583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718894    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:04:33.721583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718898    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:04:33.721583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718902    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:04:33.721583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718907    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:04:33.721583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718912    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:04:33.721583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718918    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:04:33.721583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718922    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:04:33.721583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718927    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:04:33.721583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718931    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:04:33.721583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718935    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:04:33.721583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718939    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:04:33.721583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718943    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:04:33.722377 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718948    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:04:33.722377 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718952    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:04:33.722377 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718956    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:04:33.722377 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.718960    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:04:33.722377 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721235    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:04:33.722377 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721247    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:04:33.722377 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721253    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:04:33.722377 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721258    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:04:33.722377 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721263    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:04:33.722377 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721268    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:04:33.722377 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721272    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:04:33.722377 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721277    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:04:33.722377 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721283    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:04:33.722377 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721290    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:04:33.722377 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721295    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:04:33.722377 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721299    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:04:33.722377 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721304    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:04:33.722377 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721309    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:04:33.722377 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721313    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:04:33.723213 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721317    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:04:33.723213 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721321    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:04:33.723213 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721326    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:04:33.723213 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721330    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:04:33.723213 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721335    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:04:33.723213 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721339    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:04:33.723213 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721343    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:04:33.723213 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721348    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:04:33.723213 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721352    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:04:33.723213 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721356    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:04:33.723213 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721361    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:04:33.723213 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721365    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:04:33.723213 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721369    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:04:33.723213 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721374    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:04:33.723213 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721378    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:04:33.723213 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721382    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:04:33.723213 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721386    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:04:33.723213 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721390    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:04:33.723213 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721395    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:04:33.723687 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721399    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:04:33.723687 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721403    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:04:33.723687 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721407    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:04:33.723687 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721411    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:04:33.723687 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721417    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:04:33.723687 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721421    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:04:33.723687 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721425    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:04:33.723687 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721430    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:04:33.723687 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721435    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:04:33.723687 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721440    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:04:33.723687 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721444    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:04:33.723687 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721448    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:04:33.723687 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721452    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:04:33.723687 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721456    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:04:33.723687 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721461    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:04:33.723687 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721466    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:04:33.723687 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721470    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:04:33.723687 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721474    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:04:33.723687 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721479    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:04:33.723687 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721483    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:04:33.724211 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721487    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:04:33.724211 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721492    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:04:33.724211 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721496    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:04:33.724211 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721500    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:04:33.724211 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721504    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:04:33.724211 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721509    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:04:33.724211 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721513    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:04:33.724211 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721517    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:04:33.724211 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721521    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:04:33.724211 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721525    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:04:33.724211 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721533    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:04:33.724211 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721539    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:04:33.724211 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721544    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:04:33.724211 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721549    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:04:33.724211 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721553    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:04:33.724211 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721558    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:04:33.724211 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721562    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:04:33.724211 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721567    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:04:33.724211 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721571    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:04:33.724211 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721576    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:04:33.724688 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721580    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:04:33.724688 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721584    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:04:33.724688 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721588    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:04:33.724688 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721592    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:04:33.724688 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721595    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:04:33.724688 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721599    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:04:33.724688 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721603    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:04:33.724688 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721607    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:04:33.724688 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721612    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:04:33.724688 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721616    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:04:33.724688 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721620    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:04:33.724688 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721625    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:04:33.724688 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721629    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:04:33.724688 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721633    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:04:33.724688 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721638    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:04:33.724688 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721642    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:04:33.724688 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721646    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:04:33.724688 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721650    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:04:33.724688 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721655    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:04:33.725285 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721659    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:04:33.725285 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721664    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:04:33.725285 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721668    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:04:33.725285 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721672    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:04:33.725285 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721677    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:04:33.725285 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721682    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:04:33.725285 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721686    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:04:33.725285 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721690    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:04:33.725285 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721696    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:04:33.725285 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721700    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:04:33.725285 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721704    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:04:33.725285 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721709    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:04:33.725285 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.721715    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:04:33.725285 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722544    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:04:33.725285 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722553    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:04:33.725285 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722557    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:04:33.725285 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722562    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:04:33.725285 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722566    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:04:33.725285 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722572    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:04:33.725285 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722576    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:04:33.726160 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722581    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:04:33.726160 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722585    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:04:33.726160 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722589    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:04:33.726160 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722594    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:04:33.726160 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722598    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:04:33.726160 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722602    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:04:33.726160 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722606    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:04:33.726160 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722611    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:04:33.726160 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722615    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:04:33.726160 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722619    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:04:33.726160 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722624    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:04:33.726160 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722628    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:04:33.726160 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722632    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:04:33.726160 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722636    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:04:33.726160 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722643    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:04:33.726160 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722649    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:04:33.726160 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722654    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:04:33.726160 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722658    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:04:33.726160 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722663    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:04:33.726974 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722668    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:04:33.726974 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722672    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:04:33.726974 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722676    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:04:33.726974 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722681    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:04:33.726974 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722686    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:04:33.726974 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722720    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:04:33.726974 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722724    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:04:33.726974 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722729    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:04:33.726974 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722732    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:04:33.726974 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722737    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:04:33.726974 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722741    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:04:33.726974 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722746    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:04:33.726974 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722750    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:04:33.726974 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722754    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:04:33.726974 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722758    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:04:33.726974 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722763    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:04:33.726974 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722767    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:04:33.726974 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722772    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:04:33.726974 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722776    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:04:33.726974 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722780    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:04:33.726974 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722784    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:04:33.727652 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722788    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:04:33.727652 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722793    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:04:33.727652 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722797    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:04:33.727652 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722801    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:04:33.727652 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722805    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:04:33.727652 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722809    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:04:33.727652 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722813    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:04:33.727652 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722817    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:04:33.727652 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722821    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:04:33.727652 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722826    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:04:33.727652 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722830    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:04:33.727652 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722835    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:04:33.727652 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722839    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:04:33.727652 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722843    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:04:33.727652 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722847    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:04:33.727652 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722851    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:04:33.727652 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722857    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:04:33.727652 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722862    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:04:33.727652 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722867    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:04:33.728226 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722871    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:04:33.728226 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722875    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:04:33.728226 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722880    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:04:33.728226 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722884    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:04:33.728226 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722888    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:04:33.728226 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722892    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:04:33.728226 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722897    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:04:33.728226 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722901    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:04:33.728226 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722905    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:04:33.728226 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722909    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:04:33.728226 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722914    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:04:33.728226 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722918    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:04:33.728226 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722922    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:04:33.728226 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722927    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:04:33.728226 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722932    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:04:33.728226 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722936    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:04:33.728226 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722940    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:04:33.728226 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722944    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:04:33.728226 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722948    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:04:33.728827 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722952    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:04:33.728827 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722956    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:04:33.728827 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722960    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:04:33.728827 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722964    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:04:33.728827 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722969    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:04:33.728827 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722973    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:04:33.728827 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722977    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:04:33.728827 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722981    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:04:33.728827 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722985    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:04:33.728827 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722990    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:04:33.728827 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722995    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:04:33.728827 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.722999    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:04:33.728827 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.723003    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:04:33.728827 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.723008    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:04:33.728827 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.723012    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:04:33.728827 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.723041    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:04:33.728827 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.723048    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:04:33.728827 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.723053    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:04:33.728827 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.723060    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:04:33.729437 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.723066    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:04:33.729437 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.723070    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:04:33.729437 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725020    2574 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 00:04:33.729437 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725037    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 00:04:33.729437 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725048    2574 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 00:04:33.729437 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725056    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 00:04:33.729437 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725065    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 00:04:33.729437 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725070    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 00:04:33.729437 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725077    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 00:04:33.729437 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725084    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 00:04:33.729437 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725089    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 00:04:33.729437 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725094    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 00:04:33.729437 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725101    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 00:04:33.729437 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725106    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 00:04:33.729437 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725111    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 00:04:33.729437 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725116    2574 flags.go:64] FLAG: --cgroup-root=""
Nov 03 00:04:33.729437 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725134    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 00:04:33.729437 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725140    2574 flags.go:64] FLAG: --client-ca-file=""
Nov 03 00:04:33.729437 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725144    2574 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 00:04:33.729437 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725149    2574 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 00:04:33.729437 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725155    2574 flags.go:64] FLAG: --cluster-domain=""
Nov 03 00:04:33.729437 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725160    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 00:04:33.729437 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725165    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 00:04:33.730071 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725170    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 00:04:33.730071 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725176    2574 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 00:04:33.730071 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725184    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 00:04:33.730071 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725189    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 00:04:33.730071 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725194    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 00:04:33.730071 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725199    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 00:04:33.730071 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725204    2574 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 00:04:33.730071 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725209    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 00:04:33.730071 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725214    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 00:04:33.730071 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725219    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 00:04:33.730071 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725224    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 00:04:33.730071 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725231    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 00:04:33.730071 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725236    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 00:04:33.730071 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725242    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 00:04:33.730071 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725247    2574 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 00:04:33.730071 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725252    2574 flags.go:64] FLAG: --enable-server="true"
Nov 03 00:04:33.730071 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725257    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 00:04:33.730071 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725264    2574 flags.go:64] FLAG: --event-burst="100"
Nov 03 00:04:33.730071 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725270    2574 flags.go:64] FLAG: --event-qps="50"
Nov 03 00:04:33.730071 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725275    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 00:04:33.730071 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725280    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 00:04:33.730071 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725285    2574 flags.go:64] FLAG: --eviction-hard=""
Nov 03 00:04:33.730071 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725292    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 00:04:33.730071 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725296    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 00:04:33.730727 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725301    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 00:04:33.730727 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725306    2574 flags.go:64] FLAG: --eviction-soft=""
Nov 03 00:04:33.730727 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725311    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 00:04:33.730727 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725316    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 00:04:33.730727 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725321    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 00:04:33.730727 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725326    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 00:04:33.730727 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725331    2574 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 00:04:33.730727 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725336    2574 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 00:04:33.730727 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725340    2574 flags.go:64] FLAG: --feature-gates=""
Nov 03 00:04:33.730727 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725347    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 00:04:33.730727 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725351    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 00:04:33.730727 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725358    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 00:04:33.730727 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725363    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 00:04:33.730727 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725368    2574 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 00:04:33.730727 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725373    2574 flags.go:64] FLAG: --help="false"
Nov 03 00:04:33.730727 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725378    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-133-161.ec2.internal"
Nov 03 00:04:33.730727 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725383    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 00:04:33.730727 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725388    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 00:04:33.730727 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725393    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 00:04:33.730727 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725398    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 00:04:33.730727 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725404    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 00:04:33.730727 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725409    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 00:04:33.730727 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725414    2574 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 00:04:33.731355 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725418    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 00:04:33.731355 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725423    2574 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 00:04:33.731355 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725428    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 00:04:33.731355 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725434    2574 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 00:04:33.731355 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725439    2574 flags.go:64] FLAG: --kube-reserved=""
Nov 03 00:04:33.731355 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725444    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 00:04:33.731355 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725449    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 00:04:33.731355 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725454    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 00:04:33.731355 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725459    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 00:04:33.731355 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725464    2574 flags.go:64] FLAG: --lock-file=""
Nov 03 00:04:33.731355 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725468    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 00:04:33.731355 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725473    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 00:04:33.731355 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725478    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 00:04:33.731355 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725487    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 00:04:33.731355 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725492    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 00:04:33.731355 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725497    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 00:04:33.731355 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725502    2574 flags.go:64] FLAG: --logging-format="text"
Nov 03 00:04:33.731355 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725507    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 00:04:33.731355 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725512    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 00:04:33.731355 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725517    2574 flags.go:64] FLAG: --manifest-url=""
Nov 03 00:04:33.731355 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725521    2574 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 00:04:33.731355 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725530    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 00:04:33.731355 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725535    2574 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 00:04:33.731355 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725541    2574 flags.go:64] FLAG: --max-pods="110"
Nov 03 00:04:33.731355 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725546    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725551    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725555    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725560    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725565    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725570    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725575    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725586    2574 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725591    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725596    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725601    2574 flags.go:64] FLAG: --pod-cidr=""
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725605    2574 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725610    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725615    2574 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725620    2574 flags.go:64] FLAG: --port="10250"
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725625    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725630    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fa2fd233044cd6c5"
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725635    2574 flags.go:64] FLAG: --qos-reserved=""
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725640    2574 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725645    2574 flags.go:64] FLAG: --register-node="true"
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725649    2574 flags.go:64] FLAG: --register-with-taints=""
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725656    2574 flags.go:64] FLAG: --registry-burst="10"
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725663    2574 flags.go:64] FLAG: --registry-qps="5"
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725667    2574 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725672    2574 flags.go:64] FLAG: --reserved-memory=""
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725678    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 00:04:33.731949 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725683    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 00:04:33.732576 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725689    2574 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 00:04:33.732576 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725693    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 00:04:33.732576 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725698    2574 flags.go:64] FLAG: --runonce="false"
Nov 03 00:04:33.732576 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725703    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 00:04:33.732576 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725709    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 00:04:33.732576 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725714    2574 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 00:04:33.732576 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725719    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 00:04:33.732576 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725724    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 00:04:33.732576 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725729    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 00:04:33.732576 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725734    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 00:04:33.732576 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725739    2574 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 00:04:33.732576 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725744    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 00:04:33.732576 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725748    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 00:04:33.732576 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725753    2574 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 00:04:33.732576 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725758    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 00:04:33.732576 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725763    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 00:04:33.732576 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725768    2574 flags.go:64] FLAG: --system-cgroups=""
Nov 03 00:04:33.732576 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725772    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 00:04:33.732576 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725780    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 00:04:33.732576 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725785    2574 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 00:04:33.732576 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725790    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 00:04:33.732576 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725796    2574 flags.go:64] FLAG: --tls-min-version=""
Nov 03 00:04:33.732576 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725801    2574 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 00:04:33.732576 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725806    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 00:04:33.732576 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725810    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 00:04:33.733268 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725815    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 00:04:33.733268 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725820    2574 flags.go:64] FLAG: --v="2"
Nov 03 00:04:33.733268 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725829    2574 flags.go:64] FLAG: --version="false"
Nov 03 00:04:33.733268 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725837    2574 flags.go:64] FLAG: --vmodule=""
Nov 03 00:04:33.733268 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725843    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 00:04:33.733268 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.725849    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 00:04:33.733268 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726060    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:04:33.733268 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726068    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:04:33.733268 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726072    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:04:33.733268 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726077    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:04:33.733268 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726081    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:04:33.733268 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726087    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:04:33.733268 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726091    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:04:33.733268 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726096    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:04:33.733268 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726100    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:04:33.733268 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726104    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:04:33.733268 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726110    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:04:33.733268 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726115    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:04:33.733268 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726135    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:04:33.733268 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726140    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:04:33.733268 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726144    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:04:33.733846 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726148    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:04:33.733846 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726153    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:04:33.733846 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726157    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:04:33.733846 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726161    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:04:33.733846 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726165    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:04:33.733846 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726169    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:04:33.733846 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726173    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:04:33.733846 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726177    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:04:33.733846 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726182    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:04:33.733846 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726186    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:04:33.733846 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726190    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:04:33.733846 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726195    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:04:33.733846 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726199    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:04:33.733846 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726205    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:04:33.733846 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726210    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:04:33.733846 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726214    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:04:33.733846 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726219    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:04:33.733846 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726223    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:04:33.733846 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726227    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:04:33.733846 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726231    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:04:33.734395 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726235    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:04:33.734395 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726240    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:04:33.734395 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726244    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:04:33.734395 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726250    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:04:33.734395 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726255    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:04:33.734395 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726259    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:04:33.734395 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726263    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:04:33.734395 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726269    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:04:33.734395 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726274    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:04:33.734395 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726278    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:04:33.734395 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726282    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:04:33.734395 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726288    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:04:33.734395 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726292    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:04:33.734395 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726296    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:04:33.734395 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726301    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:04:33.734395 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726305    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:04:33.734395 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726309    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:04:33.734395 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726313    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:04:33.734395 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726318    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:04:33.734395 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726322    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:04:33.734868 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726326    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:04:33.734868 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726331    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:04:33.734868 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726335    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:04:33.734868 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726339    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:04:33.734868 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726344    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:04:33.734868 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726351    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:04:33.734868 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726355    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:04:33.734868 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726359    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:04:33.734868 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726363    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:04:33.734868 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726368    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:04:33.734868 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726372    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:04:33.734868 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726376    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:04:33.734868 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726381    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:04:33.734868 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726387    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:04:33.734868 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726394    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:04:33.734868 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726399    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:04:33.734868 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726404    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:04:33.734868 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726408    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:04:33.734868 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726413    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:04:33.735420 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726419    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:04:33.735420 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726424    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:04:33.735420 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726428    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:04:33.735420 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726432    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:04:33.735420 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726437    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:04:33.735420 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726441    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:04:33.735420 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726446    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:04:33.735420 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726450    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:04:33.735420 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726454    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:04:33.735420 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726459    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:04:33.735420 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726463    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:04:33.735420 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726467    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:04:33.735420 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726471    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:04:33.735420 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726476    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:04:33.735420 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726480    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:04:33.735420 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726485    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:04:33.735420 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726489    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:04:33.735420 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726493    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:04:33.735420 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726501    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:04:33.735869 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726505    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:04:33.735869 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726509    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:04:33.735869 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726514    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:04:33.735869 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726518    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:04:33.735869 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726522    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:04:33.735869 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726526    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:04:33.735869 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726530    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:04:33.735869 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726535    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:04:33.735869 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726539    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:04:33.735869 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726544    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:04:33.735869 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726548    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:04:33.735869 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726557    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:04:33.735869 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726563    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:04:33.735869 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.726608    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 00:04:33.735869 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726665    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:04:33.736244 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726672    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:04:33.736244 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726676    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:04:33.736244 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726681    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:04:33.736244 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726686    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:04:33.736244 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726690    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:04:33.736244 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726694    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:04:33.736244 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726698    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:04:33.736244 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726706    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:04:33.736244 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726713    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:04:33.736244 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726718    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:04:33.736244 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726722    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:04:33.736244 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726727    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:04:33.736244 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726731    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:04:33.736244 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726735    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:04:33.736244 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726740    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:04:33.736244 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726744    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:04:33.736244 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726750    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:04:33.736244 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726755    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:04:33.736244 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726759    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:04:33.736716 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726763    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:04:33.736716 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726768    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:04:33.736716 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726772    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:04:33.736716 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726776    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:04:33.736716 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726780    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:04:33.736716 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726785    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:04:33.736716 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726789    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:04:33.736716 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726793    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:04:33.736716 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726798    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:04:33.736716 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726802    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:04:33.736716 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726807    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:04:33.736716 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726811    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:04:33.736716 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726817    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:04:33.736716 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726822    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:04:33.736716 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726826    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:04:33.736716 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726830    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:04:33.736716 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726835    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:04:33.736716 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726839    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:04:33.736716 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726843    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:04:33.737189 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726847    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:04:33.737189 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726851    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:04:33.737189 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726856    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:04:33.737189 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726860    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:04:33.737189 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726864    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:04:33.737189 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726869    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:04:33.737189 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726873    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:04:33.737189 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726877    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:04:33.737189 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726881    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:04:33.737189 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726886    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:04:33.737189 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726892    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:04:33.737189 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726896    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:04:33.737189 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726901    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:04:33.737189 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726905    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:04:33.737189 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726909    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:04:33.737189 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726913    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:04:33.737189 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726918    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:04:33.737189 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726922    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:04:33.737189 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726927    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:04:33.737189 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726931    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:04:33.737657 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726935    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:04:33.737657 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726940    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:04:33.737657 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726944    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:04:33.737657 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726948    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:04:33.737657 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726953    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:04:33.737657 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726959    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:04:33.737657 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726964    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:04:33.737657 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726968    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:04:33.737657 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726972    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:04:33.737657 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726976    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:04:33.737657 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726982    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:04:33.737657 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726986    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:04:33.737657 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726990    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:04:33.737657 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726994    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:04:33.737657 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.726998    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:04:33.737657 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727002    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:04:33.737657 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727007    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:04:33.737657 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727011    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:04:33.737657 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727015    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:04:33.737657 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727022    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:04:33.738145 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727027    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:04:33.738145 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727032    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:04:33.738145 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727039    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:04:33.738145 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727043    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:04:33.738145 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727047    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:04:33.738145 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727051    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:04:33.738145 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727056    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:04:33.738145 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727060    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:04:33.738145 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727065    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:04:33.738145 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727069    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:04:33.738145 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727073    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:04:33.738145 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727078    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:04:33.738145 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727082    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:04:33.738145 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727086    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:04:33.738145 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727091    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:04:33.738145 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727095    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:04:33.738145 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727099    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:04:33.738145 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727106    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:04:33.738145 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727110    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:04:33.738603 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727116    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:04:33.738603 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727135    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:04:33.738603 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727140    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:04:33.738603 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727144    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:04:33.738603 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727148    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:04:33.738603 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727152    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:04:33.738603 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727156    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:04:33.738603 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.727161    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:04:33.738603 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.735444    2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 00:04:33.738603 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.735462    2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 00:04:33.738603 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.736297    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 00:04:33.738603 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.736307    2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 00:04:33.738603 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736376    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:04:33.738603 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736382    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:04:33.738603 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736386    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:04:33.738603 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736391    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:04:33.738603 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736395    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:04:33.738603 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736399    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:04:33.738603 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736402    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:04:33.739348 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736406    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:04:33.739348 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736408    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:04:33.739348 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736411    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:04:33.739348 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736413    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:04:33.739348 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736416    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:04:33.739348 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736419    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:04:33.739348 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736421    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:04:33.739348 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736424    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:04:33.739348 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736427    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:04:33.739348 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736429    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:04:33.739348 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736432    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:04:33.739348 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736434    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:04:33.739348 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736437    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:04:33.739348 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736439    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:04:33.739348 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736442    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:04:33.739348 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736444    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:04:33.739348 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736447    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:04:33.739348 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736449    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:04:33.739348 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736452    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:04:33.739348 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736455    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:04:33.740158 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736458    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:04:33.740158 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736461    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:04:33.740158 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736469    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:04:33.740158 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736472    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:04:33.740158 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736475    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:04:33.740158 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736477    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:04:33.740158 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736480    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:04:33.740158 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736482    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:04:33.740158 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736485    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:04:33.740158 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736487    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:04:33.740158 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736490    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:04:33.740158 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736492    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:04:33.740158 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736495    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:04:33.740158 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736497    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:04:33.740158 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736500    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:04:33.740158 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736502    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:04:33.740158 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736505    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:04:33.740158 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736508    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:04:33.740158 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736510    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:04:33.740775 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736513    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:04:33.740775 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736516    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:04:33.740775 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736518    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:04:33.740775 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736521    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:04:33.740775 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736523    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:04:33.740775 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736526    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:04:33.740775 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736529    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:04:33.740775 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736532    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:04:33.740775 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736534    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:04:33.740775 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736537    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:04:33.740775 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736539    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:04:33.740775 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736542    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:04:33.740775 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736545    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:04:33.740775 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736547    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:04:33.740775 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736550    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:04:33.740775 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736553    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:04:33.740775 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736561    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:04:33.740775 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736564    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:04:33.740775 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736566    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:04:33.740775 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736569    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:04:33.741265 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736571    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:04:33.741265 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736574    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:04:33.741265 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736577    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:04:33.741265 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736579    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:04:33.741265 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736582    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:04:33.741265 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736584    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:04:33.741265 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736587    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:04:33.741265 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736589    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:04:33.741265 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736592    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:04:33.741265 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736594    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:04:33.741265 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736597    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:04:33.741265 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736599    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:04:33.741265 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736602    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:04:33.741265 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736605    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:04:33.741265 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736607    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:04:33.741265 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736610    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:04:33.741265 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736612    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:04:33.741265 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736615    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:04:33.741265 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736618    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:04:33.741711 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736622    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:04:33.741711 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736625    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:04:33.741711 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736634    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:04:33.741711 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736638    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:04:33.741711 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736642    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:04:33.741711 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736644    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:04:33.741711 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736647    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:04:33.741711 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736649    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:04:33.741711 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736652    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:04:33.741711 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736654    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:04:33.741711 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736665    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:04:33.741711 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736668    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:04:33.741711 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736670    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:04:33.741711 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736674    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:04:33.741711 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736676    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:04:33.741711 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736679    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:04:33.741711 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736681    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:04:33.741711 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736684    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:04:33.741711 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736686    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:04:33.742186 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736688    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:04:33.742186 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736691    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:04:33.742186 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.736718    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 00:04:33.742186 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736906    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:04:33.742186 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736910    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:04:33.742186 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736913    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:04:33.742186 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736916    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:04:33.742186 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736919    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:04:33.742186 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736921    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:04:33.742186 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736924    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:04:33.742186 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736926    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:04:33.742186 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736929    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:04:33.742186 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736932    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:04:33.742186 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736935    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:04:33.742186 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736937    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:04:33.742186 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736940    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:04:33.742583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736943    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:04:33.742583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736945    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:04:33.742583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736948    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:04:33.742583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736951    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:04:33.742583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736953    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:04:33.742583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736956    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:04:33.742583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736958    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:04:33.742583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736961    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:04:33.742583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736969    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:04:33.742583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736972    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:04:33.742583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736974    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:04:33.742583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736977    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:04:33.742583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736979    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:04:33.742583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736982    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:04:33.742583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736985    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:04:33.742583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736987    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:04:33.742583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736989    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:04:33.742583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736992    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:04:33.742583 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736995    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:04:33.743042 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.736997    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:04:33.743042 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737000    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:04:33.743042 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737002    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:04:33.743042 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737005    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:04:33.743042 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737007    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:04:33.743042 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737010    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:04:33.743042 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737012    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:04:33.743042 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737015    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:04:33.743042 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737018    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:04:33.743042 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737020    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:04:33.743042 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737023    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:04:33.743042 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737025    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:04:33.743042 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737028    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:04:33.743042 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737030    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:04:33.743042 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737033    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:04:33.743042 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737036    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:04:33.743042 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737038    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:04:33.743042 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737041    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:04:33.743042 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737043    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:04:33.743546 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737046    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:04:33.743546 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737048    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:04:33.743546 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737051    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:04:33.743546 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737060    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:04:33.743546 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737065    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:04:33.743546 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737068    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:04:33.743546 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737070    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:04:33.743546 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737073    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:04:33.743546 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737076    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:04:33.743546 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737079    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:04:33.743546 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737081    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:04:33.743546 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737083    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:04:33.743546 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737086    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:04:33.743546 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737088    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:04:33.743546 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737091    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:04:33.743546 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737094    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:04:33.743546 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737096    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:04:33.743546 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737099    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:04:33.743546 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737101    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:04:33.744016 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737104    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:04:33.744016 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737106    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:04:33.744016 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737109    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:04:33.744016 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737112    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:04:33.744016 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737116    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:04:33.744016 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737132    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:04:33.744016 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737144    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:04:33.744016 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737146    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:04:33.744016 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737149    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:04:33.744016 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737151    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:04:33.744016 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737154    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:04:33.744016 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737156    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:04:33.744016 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737159    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:04:33.744016 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737161    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:04:33.744016 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737164    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:04:33.744016 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737167    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:04:33.744016 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737169    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:04:33.744016 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737177    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:04:33.744016 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737180    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:04:33.744016 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737182    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:04:33.744580 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737185    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:04:33.744580 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737187    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:04:33.744580 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737190    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:04:33.744580 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737193    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:04:33.744580 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737196    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:04:33.744580 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737198    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:04:33.744580 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737201    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:04:33.744580 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737204    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:04:33.744580 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737206    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:04:33.744580 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737209    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:04:33.744580 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737212    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:04:33.744580 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737214    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:04:33.744580 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737217    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:04:33.744580 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737219    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:04:33.744580 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737221    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:04:33.744580 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737224    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:04:33.744963 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.737247    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 00:04:33.744963 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737287    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:04:33.744963 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737291    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:04:33.744963 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737294    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:04:33.744963 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737297    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:04:33.744963 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737301    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:04:33.744963 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737304    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:04:33.744963 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737307    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:04:33.744963 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737310    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:04:33.744963 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737312    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:04:33.744963 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737315    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:04:33.744963 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737317    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:04:33.744963 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737320    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:04:33.744963 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737323    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:04:33.744963 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737331    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:04:33.744963 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737334    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:04:33.745368 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737336    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:04:33.745368 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737341    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:04:33.745368 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737344    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:04:33.745368 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737346    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:04:33.745368 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737349    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:04:33.745368 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737351    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:04:33.745368 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737354    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:04:33.745368 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737357    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:04:33.745368 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737360    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:04:33.745368 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737362    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:04:33.745368 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737365    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:04:33.745368 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737367    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:04:33.745368 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737370    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:04:33.745368 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737372    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:04:33.745368 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737375    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:04:33.745368 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737377    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:04:33.745368 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737380    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:04:33.745368 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737382    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:04:33.745368 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737385    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:04:33.745368 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737387    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:04:33.745891 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737390    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:04:33.745891 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737392    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:04:33.745891 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737395    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:04:33.745891 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737397    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:04:33.745891 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737400    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:04:33.745891 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737402    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:04:33.745891 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737404    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:04:33.745891 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737407    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:04:33.745891 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737409    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:04:33.745891 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737412    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:04:33.745891 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737414    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:04:33.745891 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737425    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:04:33.745891 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737428    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:04:33.745891 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737431    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:04:33.745891 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737434    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:04:33.745891 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737437    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:04:33.745891 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737439    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:04:33.745891 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737442    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:04:33.745891 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737445    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:04:33.745891 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737448    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:04:33.746397 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737450    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:04:33.746397 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737453    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:04:33.746397 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737455    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:04:33.746397 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737458    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:04:33.746397 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737460    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:04:33.746397 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737463    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:04:33.746397 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737465    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:04:33.746397 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737468    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:04:33.746397 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737471    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:04:33.746397 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737473    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:04:33.746397 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737475    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:04:33.746397 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737478    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:04:33.746397 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737496    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:04:33.746397 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737500    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:04:33.746397 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737503    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:04:33.746397 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737506    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:04:33.746397 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737509    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:04:33.746397 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737513    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:04:33.746397 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737515    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:04:33.746397 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737518    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:04:33.746950 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737521    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:04:33.746950 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737524    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:04:33.746950 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737526    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:04:33.746950 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737529    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:04:33.746950 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737537    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:04:33.746950 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737541    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:04:33.746950 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737544    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:04:33.746950 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737547    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:04:33.746950 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737549    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:04:33.746950 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737552    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:04:33.746950 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737555    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:04:33.746950 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737558    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:04:33.746950 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737560    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:04:33.746950 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737563    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:04:33.746950 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737566    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:04:33.746950 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737568    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:04:33.746950 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737579    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:04:33.746950 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737583    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:04:33.746950 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737586    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:04:33.747451 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737589    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:04:33.747451 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737593    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:04:33.747451 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737595    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:04:33.747451 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737598    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:04:33.747451 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737601    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:04:33.747451 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737603    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:04:33.747451 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737606    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:04:33.747451 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737608    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:04:33.747451 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737611    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:04:33.747451 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737613    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:04:33.747451 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737616    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:04:33.747451 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:33.737618    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:04:33.747451 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.737680    2574 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 00:04:33.747451 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.742333    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 00:04:33.747451 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.743391    2574 server.go:1014] "Starting client certificate rotation"
Nov 03 00:04:33.747451 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.743486    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 00:04:33.747451 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.743534    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 00:04:33.747451 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.745653    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 00:04:33.747889 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.746748    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 00:04:33.759459 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.759443    2574 log.go:25] "Validated CRI v1 runtime API"
Nov 03 00:04:33.763434 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.763418    2574 log.go:25] "Validated CRI v1 image API"
Nov 03 00:04:33.764339 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.764306    2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 00:04:33.768910 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.768887    2574 fs.go:135] Filesystem UUIDs: map[4406269c-e4d8-4e91-adba-ef9b125df014:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ddbd0b69-85d2-4c2f-875f-c175681b9ede:/dev/nvme0n1p3]
Nov 03 00:04:33.768980 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.768909    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 00:04:33.775306 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.775177    2574 manager.go:217] Machine: {Timestamp:2026-11-03 00:04:33.77366741 +0000 UTC m=+0.365227958 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100076 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23955d65bf5fb8d06ae7fd50c2ea74 SystemUUID:ec23955d-65bf-5fb8-d06a-e7fd50c2ea74 BootID:85306522-6d3a-44ee-84ad-5fbca3969858 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f2:20:06:83 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f2:20:06:83 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:62:70:c2:11:83 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 00:04:33.775306 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.775305    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 00:04:33.775415 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.775385    2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 00:04:33.778803 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.778779    2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 00:04:33.778933 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.778806    2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-161.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 00:04:33.779777 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.779767    2574 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 00:04:33.779810 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.779779    2574 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 00:04:33.779810 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.779805    2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 00:04:33.779869 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.779819    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 00:04:33.781002 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.780991    2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 00:04:33.785531 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.785517    2574 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 00:04:33.785645 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.785635    2574 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 00:04:33.786490 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.786480    2574 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 00:04:33.786540 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.786493    2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 00:04:33.786540 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.786507    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 00:04:33.786540 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.786518    2574 kubelet.go:405] "Adding apiserver pod source"
Nov 03 00:04:33.786540 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.786528    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 00:04:33.786956 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.786933    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 00:04:33.791635 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.791617    2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 00:04:33.799417 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.799402    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 00:04:33.800590 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.800576    2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 00:04:33.800663 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.800617    2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 00:04:33.800810 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.800797    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 00:04:33.800867 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.800817    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 00:04:33.800867 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.800826    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 00:04:33.800867 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.800837    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 00:04:33.800867 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.800845    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 00:04:33.800867 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.800854    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 00:04:33.801602 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.801591    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 00:04:33.801658 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.801605    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 00:04:33.801658 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.801615    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 00:04:33.801658 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.801626    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 00:04:33.801658 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.801639    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 00:04:33.801658 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.801652    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 00:04:33.802438 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.802427    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 00:04:33.802497 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.802442    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 00:04:33.807845 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.807829    2574 server.go:1263] "Started kubelet"
Nov 03 00:04:33.807920 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.807894    2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 00:04:33.808010 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.807956    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 00:04:33.808051 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.808030    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 00:04:33.808671 ip-10-0-133-161 systemd[1]: Started Kubernetes Kubelet.
Nov 03 00:04:33.809552 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.809533    2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 00:04:33.810050 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.810038    2574 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 00:04:33.814199 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:33.814182    2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 00:04:33.817082 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.817063    2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 00:04:33.817187 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.817081    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 00:04:33.817187 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:33.817161    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-161.ec2.internal\" not found"
Nov 03 00:04:33.817901 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.817886    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 00:04:33.817979 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.817888    2574 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 00:04:33.817979 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.817915    2574 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 00:04:33.818080 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.818013    2574 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 00:04:33.818080 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.818022    2574 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 00:04:33.818080 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.818038    2574 factory.go:55] Registering systemd factory
Nov 03 00:04:33.818080 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.818055    2574 factory.go:223] Registration of the systemd container factory successfully
Nov 03 00:04:33.818346 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.818316    2574 factory.go:153] Registering CRI-O factory
Nov 03 00:04:33.818406 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.818348    2574 factory.go:223] Registration of the crio container factory successfully
Nov 03 00:04:33.818406 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.818399    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 00:04:33.818480 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.818422    2574 factory.go:103] Registering Raw factory
Nov 03 00:04:33.818480 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.818439    2574 manager.go:1199] Started watching for new ooms in manager
Nov 03 00:04:33.819531 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.819508    2574 manager.go:319] Starting recovery of all containers
Nov 03 00:04:33.829876 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.829860    2574 manager.go:324] Recovery completed
Nov 03 00:04:33.830482 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.830459    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 00:04:33.836194 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.836174    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:04:33.836961 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:33.836889    2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-161.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 00:04:33.836961 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.836900    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-161.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 00:04:33.838106 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:33.836896    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-161.ec2.internal.18e460275e9d1822  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-161.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-161.ec2.internal,},FirstTimestamp:2026-11-03 00:04:33.807644706 +0000 UTC m=+0.399205254,LastTimestamp:2026-11-03 00:04:33.807644706 +0000 UTC m=+0.399205254,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-161.ec2.internal,}"
Nov 03 00:04:33.839534 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.839519    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:04:33.839611 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.839546    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:04:33.839611 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.839556    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:04:33.840104 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.840087    2574 cpu_manager.go:226] "Starting" policy="none"
Nov 03 00:04:33.840104 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.840100    2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 00:04:33.840238 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.840109    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:04:33.840238 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.840114    2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 00:04:33.840808 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.840793    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:04:33.840864 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.840817    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:04:33.840864 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.840827    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:04:33.842373 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.842359    2574 policy_none.go:50] "Start"
Nov 03 00:04:33.842435 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.842377    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 00:04:33.842435 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.842390    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 00:04:33.844435 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.844424    2574 policy_none.go:44] "Start"
Nov 03 00:04:33.856249 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:33.856213    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 00:04:33.856334 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:33.856261    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 00:04:33.856334 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:33.856239    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-161.ec2.internal.18e460276083aef7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-161.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-161.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-161.ec2.internal,},FirstTimestamp:2026-11-03 00:04:33.839533815 +0000 UTC m=+0.431094362,LastTimestamp:2026-11-03 00:04:33.839533815 +0000 UTC m=+0.431094362,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-161.ec2.internal,}"
Nov 03 00:04:33.856423 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:33.856354    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-161.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 00:04:33.858653 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.858636    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hpdv7"
Nov 03 00:04:33.863590 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.863576    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hpdv7"
Nov 03 00:04:33.880395 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:33.880334    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-161.ec2.internal.18e460276083f299  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-161.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-161.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-161.ec2.internal,},FirstTimestamp:2026-11-03 00:04:33.839551129 +0000 UTC m=+0.431111676,LastTimestamp:2026-11-03 00:04:33.839551129 +0000 UTC m=+0.431111676,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-161.ec2.internal,}"
Nov 03 00:04:33.895680 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.895653    2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 00:04:33.897941 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.897930    2574 manager.go:341] "Starting Device Plugin manager"
Nov 03 00:04:33.898749 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:33.898627    2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 00:04:33.898795 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.898751    2574 server.go:87] "Starting device plugin registration server"
Nov 03 00:04:33.898962 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.898953    2574 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 00:04:33.898992 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.898965    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 00:04:33.899105 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.899086    2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 00:04:33.899212 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.899186    2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 00:04:33.899212 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.899200    2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 00:04:33.899683 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:33.899662    2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 00:04:33.899750 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:33.899720    2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-161.ec2.internal\" not found"
Nov 03 00:04:33.956778 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.956749    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 00:04:33.958652 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.957957    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 00:04:33.958652 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.957977    2574 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 00:04:33.958652 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.957995    2574 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 00:04:33.958652 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:33.958031    2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 00:04:33.970378 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.970325    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:04:33.999650 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:33.999632    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:04:34.000532 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.000515    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:04:34.000589 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.000545    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:04:34.000589 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.000555    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:04:34.000589 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.000586    2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-161.ec2.internal"
Nov 03 00:04:34.015771 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.015752    2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-161.ec2.internal"
Nov 03 00:04:34.015825 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:34.015775    2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-161.ec2.internal\": node \"ip-10-0-133-161.ec2.internal\" not found"
Nov 03 00:04:34.046044 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:34.046021    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-161.ec2.internal\" not found"
Nov 03 00:04:34.058479 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.058458    2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-161.ec2.internal"]
Nov 03 00:04:34.058538 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.058508    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:04:34.059393 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.059378    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:04:34.059447 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.059408    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:04:34.059447 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.059427    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:04:34.061549 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.061538    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:04:34.061706 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.061692    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal"
Nov 03 00:04:34.061744 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.061720    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:04:34.062320 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.062301    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:04:34.062390 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.062332    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:04:34.062390 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.062342    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:04:34.062390 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.062293    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:04:34.062495 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.062411    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:04:34.062495 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.062422    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:04:34.064322 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.064310    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-161.ec2.internal"
Nov 03 00:04:34.064366 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.064340    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:04:34.065012 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.064997    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:04:34.065091 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.065027    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:04:34.065091 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.065041    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:04:34.090558 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:34.090538    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-161.ec2.internal\" not found" node="ip-10-0-133-161.ec2.internal"
Nov 03 00:04:34.093518 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:34.093501    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-161.ec2.internal\" not found" node="ip-10-0-133-161.ec2.internal"
Nov 03 00:04:34.119908 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.119885    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b4a90a0b8aa58f888d21195c4bcfcc5c-config\") pod \"kube-apiserver-proxy-ip-10-0-133-161.ec2.internal\" (UID: \"b4a90a0b8aa58f888d21195c4bcfcc5c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-161.ec2.internal"
Nov 03 00:04:34.119995 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.119911    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/317fa55400daeb49ee1d8f32b909a6db-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal\" (UID: \"317fa55400daeb49ee1d8f32b909a6db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal"
Nov 03 00:04:34.119995 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.119928    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/317fa55400daeb49ee1d8f32b909a6db-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal\" (UID: \"317fa55400daeb49ee1d8f32b909a6db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal"
Nov 03 00:04:34.146760 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:34.146742    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-161.ec2.internal\" not found"
Nov 03 00:04:34.220065 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.220034    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/317fa55400daeb49ee1d8f32b909a6db-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal\" (UID: \"317fa55400daeb49ee1d8f32b909a6db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal"
Nov 03 00:04:34.220214 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.220074    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/317fa55400daeb49ee1d8f32b909a6db-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal\" (UID: \"317fa55400daeb49ee1d8f32b909a6db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal"
Nov 03 00:04:34.220214 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.220098    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b4a90a0b8aa58f888d21195c4bcfcc5c-config\") pod \"kube-apiserver-proxy-ip-10-0-133-161.ec2.internal\" (UID: \"b4a90a0b8aa58f888d21195c4bcfcc5c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-161.ec2.internal"
Nov 03 00:04:34.220214 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.220143    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/317fa55400daeb49ee1d8f32b909a6db-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal\" (UID: \"317fa55400daeb49ee1d8f32b909a6db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal"
Nov 03 00:04:34.220214 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.220155    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/317fa55400daeb49ee1d8f32b909a6db-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal\" (UID: \"317fa55400daeb49ee1d8f32b909a6db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal"
Nov 03 00:04:34.220214 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:34.220148    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b4a90a0b8aa58f888d21195c4bcfcc5c-config\") pod \"kube-apiserver-proxy-ip-10-0-133-161.ec2.internal\" (UID: \"b4a90a0b8aa58f888d21195c4bcfcc5c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-161.ec2.internal"
Nov 03 00:04:34.247274 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:34.247219    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-161.ec2.internal\" not found"
Nov 03 00:04:34.347985 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:34.347937    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-161.ec2.internal\" not found"
Nov 03 00:04:36.227958 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:36.227924    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal"
Nov 03 00:04:36.231043 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:36.231026    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-161.ec2.internal"
Nov 03 00:04:36.282164 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:36.282129    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-161.ec2.internal\" not found"
Nov 03 00:04:36.382730 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:36.382655    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-161.ec2.internal\" not found"
Nov 03 00:04:36.483188 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:36.483153    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-161.ec2.internal\" not found"
Nov 03 00:04:36.572966 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:36.572938    2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:04:36.577651 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:36.577635    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 00:04:36.577761 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:36.577744    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 00:04:36.577808 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:36.577779    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 00:04:36.583947 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:36.583923    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-161.ec2.internal\" not found"
Nov 03 00:04:36.684368 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:36.684311    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-161.ec2.internal\" not found"
Nov 03 00:04:36.703401 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:36.703349    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 23:59:35 +0000 UTC" deadline="2028-04-19 18:32:29.351633618 +0000 UTC"
Nov 03 00:04:36.703557 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:36.703403    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12810h27m52.648237067s"
Nov 03 00:04:36.785384 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:36.785352    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-161.ec2.internal\" not found"
Nov 03 00:04:36.799554 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:36.799521    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:04:36.802137 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:36.802121    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:04:36.802228 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:36.802149    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:04:36.802228 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:36.802160    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:04:36.821442 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:36.821414    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a90a0b8aa58f888d21195c4bcfcc5c.slice/crio-32925b0fda6815092f415eee6ae94fde3490062f322d962538f8ff5fcf9c953a WatchSource:0}: Error finding container 32925b0fda6815092f415eee6ae94fde3490062f322d962538f8ff5fcf9c953a: Status 404 returned error can't find the container with id 32925b0fda6815092f415eee6ae94fde3490062f322d962538f8ff5fcf9c953a
Nov 03 00:04:36.821676 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:36.821658    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317fa55400daeb49ee1d8f32b909a6db.slice/crio-25b8cd2ec77476e68d59d598da6f50bc0901483679ddc07512611488c8c5469e WatchSource:0}: Error finding container 25b8cd2ec77476e68d59d598da6f50bc0901483679ddc07512611488c8c5469e: Status 404 returned error can't find the container with id 25b8cd2ec77476e68d59d598da6f50bc0901483679ddc07512611488c8c5469e
Nov 03 00:04:36.826763 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:36.826748    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:04:36.885870 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:36.885849    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-161.ec2.internal\" not found"
Nov 03 00:04:36.986317 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:36.986273    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-161.ec2.internal\" not found"
Nov 03 00:04:37.086906 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:37.086881    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-161.ec2.internal\" not found"
Nov 03 00:04:37.148719 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.148695    2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:04:37.187025 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:37.187006    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-161.ec2.internal\" not found"
Nov 03 00:04:37.211525 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.211504    2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:04:37.251567 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.251522    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal"
Nov 03 00:04:37.275699 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.275681    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 00:04:37.276480 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.276468    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-161.ec2.internal"
Nov 03 00:04:37.295758 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.295736    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 00:04:37.621575 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.621505    2574 apiserver.go:52] "Watching apiserver"
Nov 03 00:04:37.651446 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.651419    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 00:04:37.659269 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.659240    2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 00:04:37.659573 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.659554    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-161.ec2.internal","openshift-multus/multus-additional-cni-plugins-bk4mx","openshift-multus/network-metrics-daemon-phwfk","kube-system/global-pull-secret-syncer-bgpx8","openshift-cluster-node-tuning-operator/tuned-7fxg5","openshift-dns/node-resolver-gtngg","openshift-image-registry/node-ca-rtkhb","openshift-multus/multus-bcgzj","openshift-network-diagnostics/network-check-target-ttc95","openshift-network-operator/iptables-alerter-f4p5q","openshift-ovn-kubernetes/ovnkube-node-xbrpr","kube-system/konnectivity-agent-ld685","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal"]
Nov 03 00:04:37.662808 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.662011    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.664240 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.664221    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:04:37.666610 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.666591    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:04:37.666710 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:37.666654    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phwfk" podUID="a173095c-7c94-4533-8529-81b464886f05"
Nov 03 00:04:37.668571 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.668552    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:04:37.670933 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.670902    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.673154 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.673134    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:04:37.673227 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:37.673203    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttc95" podUID="d683af01-29f3-4634-ae6d-3de8a5c4fd85"
Nov 03 00:04:37.675967 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.675557    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rtkhb"
Nov 03 00:04:37.676103 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.676086    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pr6ld\""
Nov 03 00:04:37.676203 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.676182    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 00:04:37.676273 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.676182    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 00:04:37.676273 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.676227    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 00:04:37.676273 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.676191    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 00:04:37.678641 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.678622    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 00:04:37.678748 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.678702    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 00:04:37.678847 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.678789    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xfsv5\""
Nov 03 00:04:37.678902 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.678873    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gtngg"
Nov 03 00:04:37.680800 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.680781    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f4p5q"
Nov 03 00:04:37.684184 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.684163    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.687174 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.687152    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ld685"
Nov 03 00:04:37.690076 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.690052    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:37.690172 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:37.690143    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgpx8" podUID="6aecdeeb-4efb-413e-a26c-9270468062a8"
Nov 03 00:04:37.698897 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.698873    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 00:04:37.699087 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.698953    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hcnvc"
Nov 03 00:04:37.707756 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.707734    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hcnvc"
Nov 03 00:04:37.753677 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.753653    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 00:04:37.763131 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.763106    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-multus-cni-dir\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.763227 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.763154    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-var-lib-cni-bin\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.763285 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.763219    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-var-lib-cni-multus\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.763340 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.763282    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-etc-kubernetes\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.763340 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.763316    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-registration-dir\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:04:37.763443 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.763346    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-sys-fs\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:04:37.763443 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.763383    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chvlq\" (UniqueName: \"kubernetes.io/projected/61d5adce-3738-4cad-aace-39d9d33a739a-kube-api-access-chvlq\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:04:37.763443 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.763428    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-run-ovn-kubernetes\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.763580 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.763486    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47ea3616-f54d-4603-a56f-5c549d66eb81-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:04:37.763580 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.763534    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-255z4\" (UniqueName: \"kubernetes.io/projected/47ea3616-f54d-4603-a56f-5c549d66eb81-kube-api-access-255z4\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:04:37.763580 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.763567    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-etc-selinux\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:04:37.763678 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.763592    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-run-netns\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.763678 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.763638    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-etc-openvswitch\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.763678 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.763661    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-cni-netd\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.763788 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.763684    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-kubernetes\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.763788 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.763735    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6aecdeeb-4efb-413e-a26c-9270468062a8-original-pull-secret\") pod \"global-pull-secret-syncer-bgpx8\" (UID: \"6aecdeeb-4efb-413e-a26c-9270468062a8\") " pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:37.763865 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.763803    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-hostroot\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.763865 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.763833    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-kubelet\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.763865 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.763856    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6cb79353-f35d-4dde-ad87-f0d03352cc49-env-overrides\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.764052 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.763900    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-sysctl-conf\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.764052 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.763983    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47ea3616-f54d-4603-a56f-5c549d66eb81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:04:37.764052 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764019    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2tzh\" (UniqueName: \"kubernetes.io/projected/ffb2a03c-414c-4d68-ab86-212c554270e2-kube-api-access-n2tzh\") pod \"node-ca-rtkhb\" (UID: \"ffb2a03c-414c-4d68-ab86-212c554270e2\") " pod="openshift-image-registry/node-ca-rtkhb"
Nov 03 00:04:37.764198 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764064    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/28e0cef7-ba0d-4546-a193-2f411c441350-hosts-file\") pod \"node-resolver-gtngg\" (UID: \"28e0cef7-ba0d-4546-a193-2f411c441350\") " pod="openshift-dns/node-resolver-gtngg"
Nov 03 00:04:37.764198 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764117    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-var-lib-kubelet\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.764198 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764165    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f59nd\" (UniqueName: \"kubernetes.io/projected/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-kube-api-access-f59nd\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.764346 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764199    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6cb79353-f35d-4dde-ad87-f0d03352cc49-ovn-node-metrics-cert\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.764346 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764245    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-run-k8s-cni-cncf-io\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.764346 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764275    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47ea3616-f54d-4603-a56f-5c549d66eb81-cnibin\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:04:37.764346 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764323    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/908118ed-04c5-483d-bb93-61b5d1116259-iptables-alerter-script\") pod \"iptables-alerter-f4p5q\" (UID: \"908118ed-04c5-483d-bb93-61b5d1116259\") " pod="openshift-network-operator/iptables-alerter-f4p5q"
Nov 03 00:04:37.764535 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764347    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-cni-binary-copy\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.764535 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764363    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-systemd-units\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.764535 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764390    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-sysconfig\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.764535 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764427    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffb2a03c-414c-4d68-ab86-212c554270e2-serviceca\") pod \"node-ca-rtkhb\" (UID: \"ffb2a03c-414c-4d68-ab86-212c554270e2\") " pod="openshift-image-registry/node-ca-rtkhb"
Nov 03 00:04:37.764535 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764471    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-run\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.764535 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764508    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-cnibin\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.764535 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764531    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-var-lib-openvswitch\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.764862 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764546    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-run-ovn\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.764862 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764568    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-var-lib-kubelet\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.764862 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764585    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47258bdf-947a-4a28-86f4-02871f64941a-konnectivity-ca\") pod \"konnectivity-agent-ld685\" (UID: \"47258bdf-947a-4a28-86f4-02871f64941a\") " pod="kube-system/konnectivity-agent-ld685"
Nov 03 00:04:37.764862 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764602    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-system-cni-dir\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.764862 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764616    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-cni-bin\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.764862 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764639    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-host\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.764862 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764654    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47ea3616-f54d-4603-a56f-5c549d66eb81-system-cni-dir\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:04:37.764862 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764670    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7dvp\" (UniqueName: \"kubernetes.io/projected/d683af01-29f3-4634-ae6d-3de8a5c4fd85-kube-api-access-k7dvp\") pod \"network-check-target-ttc95\" (UID: \"d683af01-29f3-4634-ae6d-3de8a5c4fd85\") " pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:04:37.764862 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764691    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47258bdf-947a-4a28-86f4-02871f64941a-agent-certs\") pod \"konnectivity-agent-ld685\" (UID: \"47258bdf-947a-4a28-86f4-02871f64941a\") " pod="kube-system/konnectivity-agent-ld685"
Nov 03 00:04:37.764862 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764705    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6aecdeeb-4efb-413e-a26c-9270468062a8-dbus\") pod \"global-pull-secret-syncer-bgpx8\" (UID: \"6aecdeeb-4efb-413e-a26c-9270468062a8\") " pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:37.764862 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764722    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs\") pod \"network-metrics-daemon-phwfk\" (UID: \"a173095c-7c94-4533-8529-81b464886f05\") " pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:04:37.764862 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764759    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.764862 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764775    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-lib-modules\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.764862 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764789    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffb2a03c-414c-4d68-ab86-212c554270e2-host\") pod \"node-ca-rtkhb\" (UID: \"ffb2a03c-414c-4d68-ab86-212c554270e2\") " pod="openshift-image-registry/node-ca-rtkhb"
Nov 03 00:04:37.764862 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764804    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-device-dir\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:04:37.764862 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764827    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6cb79353-f35d-4dde-ad87-f0d03352cc49-ovnkube-config\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.765542 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764842    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6cb79353-f35d-4dde-ad87-f0d03352cc49-ovnkube-script-lib\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.765542 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764857    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-run-multus-certs\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.765542 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764871    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-socket-dir\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:04:37.765542 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764885    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-sys\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.765542 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764898    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c819325-a41e-445e-858d-fa52911383fe-etc-tuned\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.765542 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764927    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c819325-a41e-445e-858d-fa52911383fe-tmp\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.765542 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764941    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-node-log\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.765542 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.764955    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e0cef7-ba0d-4546-a193-2f411c441350-tmp-dir\") pod \"node-resolver-gtngg\" (UID: \"28e0cef7-ba0d-4546-a193-2f411c441350\") " pod="openshift-dns/node-resolver-gtngg"
Nov 03 00:04:37.765542 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.765011    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/908118ed-04c5-483d-bb93-61b5d1116259-host-slash\") pod \"iptables-alerter-f4p5q\" (UID: \"908118ed-04c5-483d-bb93-61b5d1116259\") " pod="openshift-network-operator/iptables-alerter-f4p5q"
Nov 03 00:04:37.765542 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.765060    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-multus-conf-dir\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.765542 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.765081    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6aecdeeb-4efb-413e-a26c-9270468062a8-kubelet-config\") pod \"global-pull-secret-syncer-bgpx8\" (UID: \"6aecdeeb-4efb-413e-a26c-9270468062a8\") " pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:37.765542 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.765106    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47ea3616-f54d-4603-a56f-5c549d66eb81-cni-binary-copy\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:04:37.765542 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.765128    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jr76\" (UniqueName: \"kubernetes.io/projected/a173095c-7c94-4533-8529-81b464886f05-kube-api-access-8jr76\") pod \"network-metrics-daemon-phwfk\" (UID: \"a173095c-7c94-4533-8529-81b464886f05\") " pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:04:37.765542 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.765152    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-run-systemd\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.765542 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.765169    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-run-openvswitch\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.765542 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.765190    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-log-socket\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.766012 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.765208    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rwgq\" (UniqueName: \"kubernetes.io/projected/6cb79353-f35d-4dde-ad87-f0d03352cc49-kube-api-access-7rwgq\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.766012 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.765222    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-modprobe-d\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.766012 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.765237    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-sysctl-d\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.766012 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.765259    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47ea3616-f54d-4603-a56f-5c549d66eb81-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:04:37.766012 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.765276    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmwfj\" (UniqueName: \"kubernetes.io/projected/28e0cef7-ba0d-4546-a193-2f411c441350-kube-api-access-rmwfj\") pod \"node-resolver-gtngg\" (UID: \"28e0cef7-ba0d-4546-a193-2f411c441350\") " pod="openshift-dns/node-resolver-gtngg"
Nov 03 00:04:37.766012 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.765293    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsh4d\" (UniqueName: \"kubernetes.io/projected/908118ed-04c5-483d-bb93-61b5d1116259-kube-api-access-wsh4d\") pod \"iptables-alerter-f4p5q\" (UID: \"908118ed-04c5-483d-bb93-61b5d1116259\") " pod="openshift-network-operator/iptables-alerter-f4p5q"
Nov 03 00:04:37.766012 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.765307    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-multus-socket-dir-parent\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.766012 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.765327    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-run-netns\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.766012 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.765341    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-systemd\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.766012 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.765357    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52tkm\" (UniqueName: \"kubernetes.io/projected/7c819325-a41e-445e-858d-fa52911383fe-kube-api-access-52tkm\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.766012 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.765380    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-multus-daemon-config\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.766012 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.765438    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:04:37.766012 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.765488    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-slash\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.776386 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.776357    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 00:04:37.776589 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.776559    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 00:04:37.776589 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.776561    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 00:04:37.776750 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.776695    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 00:04:37.776856 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.776841    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2dzkq\""
Nov 03 00:04:37.776958 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.776944    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:04:37.777023 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.776962    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qr8z8\""
Nov 03 00:04:37.777023 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.777014    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 00:04:37.777124 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.777046    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:04:37.777124 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.777056    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 00:04:37.777124 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.777106    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tqxtt\""
Nov 03 00:04:37.777124 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.777112    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jtdbf\""
Nov 03 00:04:37.777316 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.777183    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 00:04:37.777316 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.777213    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 00:04:37.777316 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.777266    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 00:04:37.777316 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.777288    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 00:04:37.777475 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.777323    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 00:04:37.777475 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.777356    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 00:04:37.777475 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.777184    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 00:04:37.777475 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.777399    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 00:04:37.785419 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.785396    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 00:04:37.785503 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.785476    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gt4zj\""
Nov 03 00:04:37.788121 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.788003    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 00:04:37.788121 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.788042    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 00:04:37.788121 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.788056    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-nrnbx\""
Nov 03 00:04:37.788121 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.788097    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 00:04:37.788446 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.788430    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9mnsd\""
Nov 03 00:04:37.788504 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.788437    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 00:04:37.797862 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.797826    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal" event={"ID":"317fa55400daeb49ee1d8f32b909a6db","Type":"ContainerStarted","Data":"25b8cd2ec77476e68d59d598da6f50bc0901483679ddc07512611488c8c5469e"}
Nov 03 00:04:37.799063 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.799039    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-161.ec2.internal" event={"ID":"b4a90a0b8aa58f888d21195c4bcfcc5c","Type":"ContainerStarted","Data":"32925b0fda6815092f415eee6ae94fde3490062f322d962538f8ff5fcf9c953a"}
Nov 03 00:04:37.866666 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.866636    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffb2a03c-414c-4d68-ab86-212c554270e2-host\") pod \"node-ca-rtkhb\" (UID: \"ffb2a03c-414c-4d68-ab86-212c554270e2\") " pod="openshift-image-registry/node-ca-rtkhb"
Nov 03 00:04:37.866816 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.866721    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-device-dir\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:04:37.866816 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.866748    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6cb79353-f35d-4dde-ad87-f0d03352cc49-ovnkube-config\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.866816 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.866775    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffb2a03c-414c-4d68-ab86-212c554270e2-host\") pod \"node-ca-rtkhb\" (UID: \"ffb2a03c-414c-4d68-ab86-212c554270e2\") " pod="openshift-image-registry/node-ca-rtkhb"
Nov 03 00:04:37.866816 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.866796    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6cb79353-f35d-4dde-ad87-f0d03352cc49-ovnkube-script-lib\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.867039 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.866820    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-run-multus-certs\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.867039 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.866842    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-socket-dir\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:04:37.867039 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.866866    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-sys\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.867039 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.866888    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c819325-a41e-445e-858d-fa52911383fe-etc-tuned\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.867039 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.866927    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c819325-a41e-445e-858d-fa52911383fe-tmp\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.867039 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.866951    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-node-log\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.867039 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.866974    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e0cef7-ba0d-4546-a193-2f411c441350-tmp-dir\") pod \"node-resolver-gtngg\" (UID: \"28e0cef7-ba0d-4546-a193-2f411c441350\") " pod="openshift-dns/node-resolver-gtngg"
Nov 03 00:04:37.867039 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867006    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/908118ed-04c5-483d-bb93-61b5d1116259-host-slash\") pod \"iptables-alerter-f4p5q\" (UID: \"908118ed-04c5-483d-bb93-61b5d1116259\") " pod="openshift-network-operator/iptables-alerter-f4p5q"
Nov 03 00:04:37.867039 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867029    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-multus-conf-dir\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.867465 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867052    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6aecdeeb-4efb-413e-a26c-9270468062a8-kubelet-config\") pod \"global-pull-secret-syncer-bgpx8\" (UID: \"6aecdeeb-4efb-413e-a26c-9270468062a8\") " pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:37.867465 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867078    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47ea3616-f54d-4603-a56f-5c549d66eb81-cni-binary-copy\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:04:37.867465 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867156    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jr76\" (UniqueName: \"kubernetes.io/projected/a173095c-7c94-4533-8529-81b464886f05-kube-api-access-8jr76\") pod \"network-metrics-daemon-phwfk\" (UID: \"a173095c-7c94-4533-8529-81b464886f05\") " pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:04:37.867465 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867183    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-run-systemd\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.867465 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867206    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-run-openvswitch\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.867465 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867238    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-log-socket\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.867465 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867273    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rwgq\" (UniqueName: \"kubernetes.io/projected/6cb79353-f35d-4dde-ad87-f0d03352cc49-kube-api-access-7rwgq\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.867465 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867295    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-modprobe-d\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.867465 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867358    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-sysctl-d\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.867465 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867384    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6cb79353-f35d-4dde-ad87-f0d03352cc49-ovnkube-config\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.867465 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867383    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47ea3616-f54d-4603-a56f-5c549d66eb81-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:04:37.867465 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867471    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmwfj\" (UniqueName: \"kubernetes.io/projected/28e0cef7-ba0d-4546-a193-2f411c441350-kube-api-access-rmwfj\") pod \"node-resolver-gtngg\" (UID: \"28e0cef7-ba0d-4546-a193-2f411c441350\") " pod="openshift-dns/node-resolver-gtngg"
Nov 03 00:04:37.868032 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867575    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsh4d\" (UniqueName: \"kubernetes.io/projected/908118ed-04c5-483d-bb93-61b5d1116259-kube-api-access-wsh4d\") pod \"iptables-alerter-f4p5q\" (UID: \"908118ed-04c5-483d-bb93-61b5d1116259\") " pod="openshift-network-operator/iptables-alerter-f4p5q"
Nov 03 00:04:37.868032 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867623    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-multus-socket-dir-parent\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.868032 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867660    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-run-netns\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.868032 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867687    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-systemd\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.868032 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867731    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52tkm\" (UniqueName: \"kubernetes.io/projected/7c819325-a41e-445e-858d-fa52911383fe-kube-api-access-52tkm\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.868032 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867787    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-multus-daemon-config\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.868032 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867836    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:04:37.868032 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867870    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-multus-conf-dir\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.868032 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867943    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-slash\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.868032 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867979    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-run-openvswitch\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.868032 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867988    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-device-dir\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:04:37.868032 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.868036    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-run-multus-certs\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.868577 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.868044    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6aecdeeb-4efb-413e-a26c-9270468062a8-kubelet-config\") pod \"global-pull-secret-syncer-bgpx8\" (UID: \"6aecdeeb-4efb-413e-a26c-9270468062a8\") " pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:37.868577 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.868076    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47ea3616-f54d-4603-a56f-5c549d66eb81-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:04:37.868577 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.868149    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6cb79353-f35d-4dde-ad87-f0d03352cc49-ovnkube-script-lib\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.868577 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.868150    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-socket-dir\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:04:37.868577 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.868200    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-sys\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.868577 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.868212    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-log-socket\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.868577 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.868489    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47ea3616-f54d-4603-a56f-5c549d66eb81-cni-binary-copy\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:04:37.868577 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.868562    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-run-systemd\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.868968 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.868578    2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 00:04:37.868968 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.868612    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-systemd\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.868968 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.868663    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/908118ed-04c5-483d-bb93-61b5d1116259-host-slash\") pod \"iptables-alerter-f4p5q\" (UID: \"908118ed-04c5-483d-bb93-61b5d1116259\") " pod="openshift-network-operator/iptables-alerter-f4p5q"
Nov 03 00:04:37.868968 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.867874    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-slash\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.868968 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.868697    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e0cef7-ba0d-4546-a193-2f411c441350-tmp-dir\") pod \"node-resolver-gtngg\" (UID: \"28e0cef7-ba0d-4546-a193-2f411c441350\") " pod="openshift-dns/node-resolver-gtngg"
Nov 03 00:04:37.868968 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.868732    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-multus-cni-dir\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.868968 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.868771    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-var-lib-cni-bin\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.868968 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.868809    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-var-lib-cni-multus\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.868968 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.868839    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-etc-kubernetes\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.868968 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.868851    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-node-log\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.868968 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.868864    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-registration-dir\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:04:37.868968 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.868888    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-sys-fs\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:04:37.868968 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.868904    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-var-lib-cni-bin\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.868968 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.868939    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chvlq\" (UniqueName: \"kubernetes.io/projected/61d5adce-3738-4cad-aace-39d9d33a739a-kube-api-access-chvlq\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:04:37.868968 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.868950    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-etc-kubernetes\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.868968 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.868957    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-multus-socket-dir-parent\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.869472 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.868992    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-multus-cni-dir\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.869472 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.869003    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-run-netns\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.869472 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.869034    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-run-ovn-kubernetes\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.869472 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.869022    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-registration-dir\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:04:37.869472 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.869080    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-run-ovn-kubernetes\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.869472 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.869110    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-sys-fs\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:04:37.869472 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.869262    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-sysctl-d\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.869472 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.869307    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-var-lib-cni-multus\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.869472 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.869342    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:04:37.869472 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.869420    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-modprobe-d\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.869903 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.869522    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47ea3616-f54d-4603-a56f-5c549d66eb81-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:04:37.869903 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.869549    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-255z4\" (UniqueName: \"kubernetes.io/projected/47ea3616-f54d-4603-a56f-5c549d66eb81-kube-api-access-255z4\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:04:37.869903 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.869567    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-etc-selinux\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:04:37.869903 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.869582    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-run-netns\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.869903 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.869602    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-etc-openvswitch\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.869903 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.869624    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-cni-netd\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.869903 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.869646    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-kubernetes\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.869903 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.869676    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-run-netns\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.870345 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.869670    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6aecdeeb-4efb-413e-a26c-9270468062a8-original-pull-secret\") pod \"global-pull-secret-syncer-bgpx8\" (UID: \"6aecdeeb-4efb-413e-a26c-9270468062a8\") " pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:37.870431 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.870385    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-hostroot\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.870489 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.870430    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-kubelet\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.870489 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.870464    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6cb79353-f35d-4dde-ad87-f0d03352cc49-env-overrides\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.870579 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.870495    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-sysctl-conf\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.870579 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.870519    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47ea3616-f54d-4603-a56f-5c549d66eb81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:04:37.870693 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.870675    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2tzh\" (UniqueName: \"kubernetes.io/projected/ffb2a03c-414c-4d68-ab86-212c554270e2-kube-api-access-n2tzh\") pod \"node-ca-rtkhb\" (UID: \"ffb2a03c-414c-4d68-ab86-212c554270e2\") " pod="openshift-image-registry/node-ca-rtkhb"
Nov 03 00:04:37.870755 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.870710    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/28e0cef7-ba0d-4546-a193-2f411c441350-hosts-file\") pod \"node-resolver-gtngg\" (UID: \"28e0cef7-ba0d-4546-a193-2f411c441350\") " pod="openshift-dns/node-resolver-gtngg"
Nov 03 00:04:37.870755 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.870735    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-var-lib-kubelet\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.870844 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.870763    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f59nd\" (UniqueName: \"kubernetes.io/projected/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-kube-api-access-f59nd\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.870844 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.870785    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6cb79353-f35d-4dde-ad87-f0d03352cc49-ovn-node-metrics-cert\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.870995 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.870974    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-run-k8s-cni-cncf-io\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.871069 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.871019    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47ea3616-f54d-4603-a56f-5c549d66eb81-cnibin\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:04:37.871069 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.871052    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/908118ed-04c5-483d-bb93-61b5d1116259-iptables-alerter-script\") pod \"iptables-alerter-f4p5q\" (UID: \"908118ed-04c5-483d-bb93-61b5d1116259\") " pod="openshift-network-operator/iptables-alerter-f4p5q"
Nov 03 00:04:37.871174 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.871084    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-cni-binary-copy\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.871174 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.871119    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-systemd-units\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.871329 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.871150    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-sysconfig\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.871384 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.871352    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffb2a03c-414c-4d68-ab86-212c554270e2-serviceca\") pod \"node-ca-rtkhb\" (UID: \"ffb2a03c-414c-4d68-ab86-212c554270e2\") " pod="openshift-image-registry/node-ca-rtkhb"
Nov 03 00:04:37.871435 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.871383    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-run\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.871435 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.871407    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-cnibin\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.871530 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.871437    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-var-lib-openvswitch\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.871530 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.871468    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-run-ovn\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.871646 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.871588    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-var-lib-kubelet\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.871646 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.871617    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47258bdf-947a-4a28-86f4-02871f64941a-konnectivity-ca\") pod \"konnectivity-agent-ld685\" (UID: \"47258bdf-947a-4a28-86f4-02871f64941a\") " pod="kube-system/konnectivity-agent-ld685"
Nov 03 00:04:37.871646 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.871639    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-system-cni-dir\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.871771 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.871664    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-cni-bin\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.871771 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.871690    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-host\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.871935 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.871864    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47ea3616-f54d-4603-a56f-5c549d66eb81-system-cni-dir\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:04:37.871935 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.871899    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7dvp\" (UniqueName: \"kubernetes.io/projected/d683af01-29f3-4634-ae6d-3de8a5c4fd85-kube-api-access-k7dvp\") pod \"network-check-target-ttc95\" (UID: \"d683af01-29f3-4634-ae6d-3de8a5c4fd85\") " pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:04:37.872042 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.871946    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47258bdf-947a-4a28-86f4-02871f64941a-agent-certs\") pod \"konnectivity-agent-ld685\" (UID: \"47258bdf-947a-4a28-86f4-02871f64941a\") " pod="kube-system/konnectivity-agent-ld685"
Nov 03 00:04:37.872042 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.871975    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6aecdeeb-4efb-413e-a26c-9270468062a8-dbus\") pod \"global-pull-secret-syncer-bgpx8\" (UID: \"6aecdeeb-4efb-413e-a26c-9270468062a8\") " pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:37.872042 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.872006    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs\") pod \"network-metrics-daemon-phwfk\" (UID: \"a173095c-7c94-4533-8529-81b464886f05\") " pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:04:37.872193 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.872043    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.872193 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.872169    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-lib-modules\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.875334 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.875308    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-multus-daemon-config\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.875441 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.875353    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47ea3616-f54d-4603-a56f-5c549d66eb81-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:04:37.875441 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.875390    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c819325-a41e-445e-858d-fa52911383fe-tmp\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.875555 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.875501    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c819325-a41e-445e-858d-fa52911383fe-etc-tuned\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.875609 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.875594    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-run\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.875952 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.875933    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47258bdf-947a-4a28-86f4-02871f64941a-konnectivity-ca\") pod \"konnectivity-agent-ld685\" (UID: \"47258bdf-947a-4a28-86f4-02871f64941a\") " pod="kube-system/konnectivity-agent-ld685"
Nov 03 00:04:37.876025 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.875987    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/908118ed-04c5-483d-bb93-61b5d1116259-iptables-alerter-script\") pod \"iptables-alerter-f4p5q\" (UID: \"908118ed-04c5-483d-bb93-61b5d1116259\") " pod="openshift-network-operator/iptables-alerter-f4p5q"
Nov 03 00:04:37.876088 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.876043    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-cnibin\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.876088 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.876061    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-etc-openvswitch\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.876189 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.876073    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-cni-binary-copy\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.876189 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.876127    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-systemd-units\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.876189 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.876160    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-var-lib-openvswitch\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.876335 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.876201    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-sysconfig\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.876335 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.876199    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-hostroot\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.876335 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.876214    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-run-ovn\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.876335 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.876291    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-kubernetes\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.876519 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.876373    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-kubelet\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.876519 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.876382    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-system-cni-dir\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.876519 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.876421    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-cni-bin\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.876519 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.876484    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-host\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.876707 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.876525    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47ea3616-f54d-4603-a56f-5c549d66eb81-system-cni-dir\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:04:37.876707 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.876585    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6aecdeeb-4efb-413e-a26c-9270468062a8-dbus\") pod \"global-pull-secret-syncer-bgpx8\" (UID: \"6aecdeeb-4efb-413e-a26c-9270468062a8\") " pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:37.876811 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.876742    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47ea3616-f54d-4603-a56f-5c549d66eb81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:04:37.877046 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.877027    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6cb79353-f35d-4dde-ad87-f0d03352cc49-env-overrides\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.877130 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.877034    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/28e0cef7-ba0d-4546-a193-2f411c441350-hosts-file\") pod \"node-resolver-gtngg\" (UID: \"28e0cef7-ba0d-4546-a193-2f411c441350\") " pod="openshift-dns/node-resolver-gtngg"
Nov 03 00:04:37.877130 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.877086    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-var-lib-kubelet\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.877363 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.877346    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.877476 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.877450    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-etc-selinux\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:04:37.877570 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:37.877557    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:37.877710 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:37.877699    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aecdeeb-4efb-413e-a26c-9270468062a8-original-pull-secret podName:6aecdeeb-4efb-413e-a26c-9270468062a8 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:38.377677527 +0000 UTC m=+3.135450095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6aecdeeb-4efb-413e-a26c-9270468062a8-original-pull-secret") pod "global-pull-secret-syncer-bgpx8" (UID: "6aecdeeb-4efb-413e-a26c-9270468062a8") : object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:37.877976 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.877953    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-run-k8s-cni-cncf-io\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.878044 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.878007    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47ea3616-f54d-4603-a56f-5c549d66eb81-cnibin\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:04:37.878108 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.878050    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-cni-netd\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.878931 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:37.878746    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:37.878931 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.878873    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-sysctl-conf\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.879947 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.879902    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6cb79353-f35d-4dde-ad87-f0d03352cc49-ovn-node-metrics-cert\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.880039 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.879987    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffb2a03c-414c-4d68-ab86-212c554270e2-serviceca\") pod \"node-ca-rtkhb\" (UID: \"ffb2a03c-414c-4d68-ab86-212c554270e2\") " pod="openshift-image-registry/node-ca-rtkhb"
Nov 03 00:04:37.880039 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:37.880012    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs podName:a173095c-7c94-4533-8529-81b464886f05 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:38.379980654 +0000 UTC m=+3.137753222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs") pod "network-metrics-daemon-phwfk" (UID: "a173095c-7c94-4533-8529-81b464886f05") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:37.880201 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.880163    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47258bdf-947a-4a28-86f4-02871f64941a-agent-certs\") pod \"konnectivity-agent-ld685\" (UID: \"47258bdf-947a-4a28-86f4-02871f64941a\") " pod="kube-system/konnectivity-agent-ld685"
Nov 03 00:04:37.881202 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.880929    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-var-lib-kubelet\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.881202 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.880988    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-lib-modules\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.881848 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.881489    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:04:37.887185 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.887163    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsh4d\" (UniqueName: \"kubernetes.io/projected/908118ed-04c5-483d-bb93-61b5d1116259-kube-api-access-wsh4d\") pod \"iptables-alerter-f4p5q\" (UID: \"908118ed-04c5-483d-bb93-61b5d1116259\") " pod="openshift-network-operator/iptables-alerter-f4p5q"
Nov 03 00:04:37.887382 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.887363    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chvlq\" (UniqueName: \"kubernetes.io/projected/61d5adce-3738-4cad-aace-39d9d33a739a-kube-api-access-chvlq\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:04:37.887662 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.887644    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmwfj\" (UniqueName: \"kubernetes.io/projected/28e0cef7-ba0d-4546-a193-2f411c441350-kube-api-access-rmwfj\") pod \"node-resolver-gtngg\" (UID: \"28e0cef7-ba0d-4546-a193-2f411c441350\") " pod="openshift-dns/node-resolver-gtngg"
Nov 03 00:04:37.888357 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.888334    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rwgq\" (UniqueName: \"kubernetes.io/projected/6cb79353-f35d-4dde-ad87-f0d03352cc49-kube-api-access-7rwgq\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:37.888990 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.888949    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jr76\" (UniqueName: \"kubernetes.io/projected/a173095c-7c94-4533-8529-81b464886f05-kube-api-access-8jr76\") pod \"network-metrics-daemon-phwfk\" (UID: \"a173095c-7c94-4533-8529-81b464886f05\") " pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:04:37.892815 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:37.892699    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 00:04:37.892815 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:37.892721    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 00:04:37.892815 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:37.892734    2574 projected.go:196] Error preparing data for projected volume kube-api-access-k7dvp for pod openshift-network-diagnostics/network-check-target-ttc95: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:37.892815 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:37.892782    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d683af01-29f3-4634-ae6d-3de8a5c4fd85-kube-api-access-k7dvp podName:d683af01-29f3-4634-ae6d-3de8a5c4fd85 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:38.392767583 +0000 UTC m=+3.150540154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k7dvp" (UniqueName: "kubernetes.io/projected/d683af01-29f3-4634-ae6d-3de8a5c4fd85-kube-api-access-k7dvp") pod "network-check-target-ttc95" (UID: "d683af01-29f3-4634-ae6d-3de8a5c4fd85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:37.893548 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.893525    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-255z4\" (UniqueName: \"kubernetes.io/projected/47ea3616-f54d-4603-a56f-5c549d66eb81-kube-api-access-255z4\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:04:37.893937 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.893904    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52tkm\" (UniqueName: \"kubernetes.io/projected/7c819325-a41e-445e-858d-fa52911383fe-kube-api-access-52tkm\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:37.894133 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.894114    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2tzh\" (UniqueName: \"kubernetes.io/projected/ffb2a03c-414c-4d68-ab86-212c554270e2-kube-api-access-n2tzh\") pod \"node-ca-rtkhb\" (UID: \"ffb2a03c-414c-4d68-ab86-212c554270e2\") " pod="openshift-image-registry/node-ca-rtkhb"
Nov 03 00:04:37.895296 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.895278    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f59nd\" (UniqueName: \"kubernetes.io/projected/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-kube-api-access-f59nd\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.979051 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:37.979027    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bcgzj"
Nov 03 00:04:37.989549 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:37.989522    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015c1f9d_d4c4_4749_978a_3cd2166c7bf8.slice/crio-84cd2d9a687e85e59dd66448d9c7f03e7d89bf34dab38621ced8b345f589eeb0 WatchSource:0}: Error finding container 84cd2d9a687e85e59dd66448d9c7f03e7d89bf34dab38621ced8b345f589eeb0: Status 404 returned error can't find the container with id 84cd2d9a687e85e59dd66448d9c7f03e7d89bf34dab38621ced8b345f589eeb0
Nov 03 00:04:38.009978 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.009955    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:04:38.015101 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.015074    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:04:38.019753 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:38.019725    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ea3616_f54d_4603_a56f_5c549d66eb81.slice/crio-8ce6a3437092f12bae141073692ecf67563bf2879195884d2fabe043160a95ec WatchSource:0}: Error finding container 8ce6a3437092f12bae141073692ecf67563bf2879195884d2fabe043160a95ec: Status 404 returned error can't find the container with id 8ce6a3437092f12bae141073692ecf67563bf2879195884d2fabe043160a95ec
Nov 03 00:04:38.020650 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.020507    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rtkhb"
Nov 03 00:04:38.023357 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.023217    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gtngg"
Nov 03 00:04:38.025408 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:38.025259    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d5adce_3738_4cad_aace_39d9d33a739a.slice/crio-dce57bb4de821416aa2640454e1413069f4756379b69eb399c55299f4d1d1219 WatchSource:0}: Error finding container dce57bb4de821416aa2640454e1413069f4756379b69eb399c55299f4d1d1219: Status 404 returned error can't find the container with id dce57bb4de821416aa2640454e1413069f4756379b69eb399c55299f4d1d1219
Nov 03 00:04:38.033544 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:38.033518    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb2a03c_414c_4d68_ab86_212c554270e2.slice/crio-f91b5735012147ec87a0fee6fe44f7791385e421c0d95eb9e70b9d5d8ea000f2 WatchSource:0}: Error finding container f91b5735012147ec87a0fee6fe44f7791385e421c0d95eb9e70b9d5d8ea000f2: Status 404 returned error can't find the container with id f91b5735012147ec87a0fee6fe44f7791385e421c0d95eb9e70b9d5d8ea000f2
Nov 03 00:04:38.034482 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.034310    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:38.035377 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:38.035343    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e0cef7_ba0d_4546_a193_2f411c441350.slice/crio-fb6385809f85d1b7277cd035d86d0eb1b4ed29b00d661d6406ffe3bcf1d7ae20 WatchSource:0}: Error finding container fb6385809f85d1b7277cd035d86d0eb1b4ed29b00d661d6406ffe3bcf1d7ae20: Status 404 returned error can't find the container with id fb6385809f85d1b7277cd035d86d0eb1b4ed29b00d661d6406ffe3bcf1d7ae20
Nov 03 00:04:38.036473 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.035962    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f4p5q"
Nov 03 00:04:38.038998 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.038713    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:04:38.044521 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.044387    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ld685"
Nov 03 00:04:38.045835 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:38.045812    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb79353_f35d_4dde_ad87_f0d03352cc49.slice/crio-a22d53dec8b80ea7bce25b6229cff9d1c7b321b62c8adfb26174dc5682256fa4 WatchSource:0}: Error finding container a22d53dec8b80ea7bce25b6229cff9d1c7b321b62c8adfb26174dc5682256fa4: Status 404 returned error can't find the container with id a22d53dec8b80ea7bce25b6229cff9d1c7b321b62c8adfb26174dc5682256fa4
Nov 03 00:04:38.056330 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:04:38.056307    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47258bdf_947a_4a28_86f4_02871f64941a.slice/crio-1ffb3ce37f4ea01beca43eaf7f71968d3f6bf22a280aa4e6de499ade77a4ab42 WatchSource:0}: Error finding container 1ffb3ce37f4ea01beca43eaf7f71968d3f6bf22a280aa4e6de499ade77a4ab42: Status 404 returned error can't find the container with id 1ffb3ce37f4ea01beca43eaf7f71968d3f6bf22a280aa4e6de499ade77a4ab42
Nov 03 00:04:38.478438 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.477760    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7dvp\" (UniqueName: \"kubernetes.io/projected/d683af01-29f3-4634-ae6d-3de8a5c4fd85-kube-api-access-k7dvp\") pod \"network-check-target-ttc95\" (UID: \"d683af01-29f3-4634-ae6d-3de8a5c4fd85\") " pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:04:38.478438 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.478016    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs\") pod \"network-metrics-daemon-phwfk\" (UID: \"a173095c-7c94-4533-8529-81b464886f05\") " pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:04:38.478438 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:38.478033    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 00:04:38.478438 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:38.478051    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 00:04:38.478438 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:38.478061    2574 projected.go:196] Error preparing data for projected volume kube-api-access-k7dvp for pod openshift-network-diagnostics/network-check-target-ttc95: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:38.478438 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.478083    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6aecdeeb-4efb-413e-a26c-9270468062a8-original-pull-secret\") pod \"global-pull-secret-syncer-bgpx8\" (UID: \"6aecdeeb-4efb-413e-a26c-9270468062a8\") " pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:38.478438 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:38.478107    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d683af01-29f3-4634-ae6d-3de8a5c4fd85-kube-api-access-k7dvp podName:d683af01-29f3-4634-ae6d-3de8a5c4fd85 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:39.478091022 +0000 UTC m=+4.235863574 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7dvp" (UniqueName: "kubernetes.io/projected/d683af01-29f3-4634-ae6d-3de8a5c4fd85-kube-api-access-k7dvp") pod "network-check-target-ttc95" (UID: "d683af01-29f3-4634-ae6d-3de8a5c4fd85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:38.478438 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:38.478178    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:38.478438 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:38.478340    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:38.478438 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:38.478389    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs podName:a173095c-7c94-4533-8529-81b464886f05 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:39.478373011 +0000 UTC m=+4.236145582 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs") pod "network-metrics-daemon-phwfk" (UID: "a173095c-7c94-4533-8529-81b464886f05") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:38.479836 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:38.479810    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aecdeeb-4efb-413e-a26c-9270468062a8-original-pull-secret podName:6aecdeeb-4efb-413e-a26c-9270468062a8 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:39.479788355 +0000 UTC m=+4.237560930 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6aecdeeb-4efb-413e-a26c-9270468062a8-original-pull-secret") pod "global-pull-secret-syncer-bgpx8" (UID: "6aecdeeb-4efb-413e-a26c-9270468062a8") : object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:38.711232 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.711117    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 23:59:37 +0000 UTC" deadline="2028-06-18 20:26:47.704105474 +0000 UTC"
Nov 03 00:04:38.711232 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.711168    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14252h22m8.992941172s"
Nov 03 00:04:38.793450 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.792983    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:38.793450 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:38.793103    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgpx8" podUID="6aecdeeb-4efb-413e-a26c-9270468062a8"
Nov 03 00:04:38.804790 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.804736    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ld685" event={"ID":"47258bdf-947a-4a28-86f4-02871f64941a","Type":"ContainerStarted","Data":"1ffb3ce37f4ea01beca43eaf7f71968d3f6bf22a280aa4e6de499ade77a4ab42"}
Nov 03 00:04:38.814435 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.814355    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f4p5q" event={"ID":"908118ed-04c5-483d-bb93-61b5d1116259","Type":"ContainerStarted","Data":"ddf0b02c07944ca9b610fb313cc5817c926a312bebea337199d8e15ec25e4276"}
Nov 03 00:04:38.820117 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.820037    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rtkhb" event={"ID":"ffb2a03c-414c-4d68-ab86-212c554270e2","Type":"ContainerStarted","Data":"f91b5735012147ec87a0fee6fe44f7791385e421c0d95eb9e70b9d5d8ea000f2"}
Nov 03 00:04:38.824483 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.824394    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4" event={"ID":"61d5adce-3738-4cad-aace-39d9d33a739a","Type":"ContainerStarted","Data":"dce57bb4de821416aa2640454e1413069f4756379b69eb399c55299f4d1d1219"}
Nov 03 00:04:38.829064 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.829000    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bcgzj" event={"ID":"015c1f9d-d4c4-4749-978a-3cd2166c7bf8","Type":"ContainerStarted","Data":"84cd2d9a687e85e59dd66448d9c7f03e7d89bf34dab38621ced8b345f589eeb0"}
Nov 03 00:04:38.832879 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.832156    2574 generic.go:356] "Generic (PLEG): container finished" podID="317fa55400daeb49ee1d8f32b909a6db" containerID="2c5b611bbafcad9f4d61ebaad27fd448fe8c66a6ae02abca20f4216e2f97c366" exitCode=0
Nov 03 00:04:38.832879 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.832224    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal" event={"ID":"317fa55400daeb49ee1d8f32b909a6db","Type":"ContainerDied","Data":"2c5b611bbafcad9f4d61ebaad27fd448fe8c66a6ae02abca20f4216e2f97c366"}
Nov 03 00:04:38.842253 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.841833    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-161.ec2.internal" event={"ID":"b4a90a0b8aa58f888d21195c4bcfcc5c","Type":"ContainerStarted","Data":"302825afb86b779681d3c71625afa8883a299c5f3648182bc72a1ed7ca77c2ad"}
Nov 03 00:04:38.852042 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.851821    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7fxg5" event={"ID":"7c819325-a41e-445e-858d-fa52911383fe","Type":"ContainerStarted","Data":"7b76785bff734f1e27e7e506789c237880548fd249d51adab7e7e6696f392bc9"}
Nov 03 00:04:38.856629 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.856537    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr" event={"ID":"6cb79353-f35d-4dde-ad87-f0d03352cc49","Type":"ContainerStarted","Data":"a22d53dec8b80ea7bce25b6229cff9d1c7b321b62c8adfb26174dc5682256fa4"}
Nov 03 00:04:38.859599 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.859548    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gtngg" event={"ID":"28e0cef7-ba0d-4546-a193-2f411c441350","Type":"ContainerStarted","Data":"fb6385809f85d1b7277cd035d86d0eb1b4ed29b00d661d6406ffe3bcf1d7ae20"}
Nov 03 00:04:38.861430 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.861372    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bk4mx" event={"ID":"47ea3616-f54d-4603-a56f-5c549d66eb81","Type":"ContainerStarted","Data":"8ce6a3437092f12bae141073692ecf67563bf2879195884d2fabe043160a95ec"}
Nov 03 00:04:38.986747 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:38.986615    2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:04:39.487191 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:39.487149    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6aecdeeb-4efb-413e-a26c-9270468062a8-original-pull-secret\") pod \"global-pull-secret-syncer-bgpx8\" (UID: \"6aecdeeb-4efb-413e-a26c-9270468062a8\") " pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:39.487375 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:39.487223    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7dvp\" (UniqueName: \"kubernetes.io/projected/d683af01-29f3-4634-ae6d-3de8a5c4fd85-kube-api-access-k7dvp\") pod \"network-check-target-ttc95\" (UID: \"d683af01-29f3-4634-ae6d-3de8a5c4fd85\") " pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:04:39.487375 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:39.487253    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs\") pod \"network-metrics-daemon-phwfk\" (UID: \"a173095c-7c94-4533-8529-81b464886f05\") " pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:04:39.487495 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:39.487399    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:39.487495 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:39.487461    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs podName:a173095c-7c94-4533-8529-81b464886f05 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:41.487442722 +0000 UTC m=+6.245215295 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs") pod "network-metrics-daemon-phwfk" (UID: "a173095c-7c94-4533-8529-81b464886f05") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:39.487887 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:39.487865    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:39.487998 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:39.487942    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aecdeeb-4efb-413e-a26c-9270468062a8-original-pull-secret podName:6aecdeeb-4efb-413e-a26c-9270468062a8 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:41.487926003 +0000 UTC m=+6.245698555 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6aecdeeb-4efb-413e-a26c-9270468062a8-original-pull-secret") pod "global-pull-secret-syncer-bgpx8" (UID: "6aecdeeb-4efb-413e-a26c-9270468062a8") : object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:39.488066 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:39.488026    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 00:04:39.488066 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:39.488040    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 00:04:39.488066 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:39.488052    2574 projected.go:196] Error preparing data for projected volume kube-api-access-k7dvp for pod openshift-network-diagnostics/network-check-target-ttc95: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:39.488207 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:39.488087    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d683af01-29f3-4634-ae6d-3de8a5c4fd85-kube-api-access-k7dvp podName:d683af01-29f3-4634-ae6d-3de8a5c4fd85 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:41.48807666 +0000 UTC m=+6.245849225 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7dvp" (UniqueName: "kubernetes.io/projected/d683af01-29f3-4634-ae6d-3de8a5c4fd85-kube-api-access-k7dvp") pod "network-check-target-ttc95" (UID: "d683af01-29f3-4634-ae6d-3de8a5c4fd85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:39.711545 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:39.711497    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 23:59:37 +0000 UTC" deadline="2028-04-05 17:48:53.714309438 +0000 UTC"
Nov 03 00:04:39.711545 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:39.711542    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12473h44m14.002772043s"
Nov 03 00:04:39.793146 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:39.793026    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:04:39.793278 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:39.793189    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phwfk" podUID="a173095c-7c94-4533-8529-81b464886f05"
Nov 03 00:04:39.793278 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:39.793234    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:04:39.793345 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:39.793309    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttc95" podUID="d683af01-29f3-4634-ae6d-3de8a5c4fd85"
Nov 03 00:04:39.884437 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:39.884307    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal" event={"ID":"317fa55400daeb49ee1d8f32b909a6db","Type":"ContainerStarted","Data":"5d03f4f00e77eb15f0a30517b16c8fd85e30f4edb20925739e071fc619db565b"}
Nov 03 00:04:39.923557 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:39.923122    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-161.ec2.internal" podStartSLOduration=2.923099808 podStartE2EDuration="2.923099808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:38.904305242 +0000 UTC m=+3.662077816" watchObservedRunningTime="2026-11-03 00:04:39.923099808 +0000 UTC m=+4.680872383"
Nov 03 00:04:40.792992 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:40.792956    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:40.793463 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:40.793102    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgpx8" podUID="6aecdeeb-4efb-413e-a26c-9270468062a8"
Nov 03 00:04:40.900081 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:40.899372    2574 generic.go:356] "Generic (PLEG): container finished" podID="47ea3616-f54d-4603-a56f-5c549d66eb81" containerID="7ce414a2fe5b47ffe1fb85420f1a1a2ee9b0658559c4aa64acfdaea98ed6cdcf" exitCode=0
Nov 03 00:04:40.900081 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:40.900013    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bk4mx" event={"ID":"47ea3616-f54d-4603-a56f-5c549d66eb81","Type":"ContainerDied","Data":"7ce414a2fe5b47ffe1fb85420f1a1a2ee9b0658559c4aa64acfdaea98ed6cdcf"}
Nov 03 00:04:40.964266 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:40.964201    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal" podStartSLOduration=3.964179181 podStartE2EDuration="3.964179181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:39.923541017 +0000 UTC m=+4.681313588" watchObservedRunningTime="2026-11-03 00:04:40.964179181 +0000 UTC m=+5.721951759"
Nov 03 00:04:41.506666 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:41.506624    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7dvp\" (UniqueName: \"kubernetes.io/projected/d683af01-29f3-4634-ae6d-3de8a5c4fd85-kube-api-access-k7dvp\") pod \"network-check-target-ttc95\" (UID: \"d683af01-29f3-4634-ae6d-3de8a5c4fd85\") " pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:04:41.506847 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:41.506680    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs\") pod \"network-metrics-daemon-phwfk\" (UID: \"a173095c-7c94-4533-8529-81b464886f05\") " pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:04:41.506847 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:41.506752    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6aecdeeb-4efb-413e-a26c-9270468062a8-original-pull-secret\") pod \"global-pull-secret-syncer-bgpx8\" (UID: \"6aecdeeb-4efb-413e-a26c-9270468062a8\") " pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:41.506984 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:41.506883    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:41.506984 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:41.506959    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aecdeeb-4efb-413e-a26c-9270468062a8-original-pull-secret podName:6aecdeeb-4efb-413e-a26c-9270468062a8 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:45.506938324 +0000 UTC m=+10.264710895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6aecdeeb-4efb-413e-a26c-9270468062a8-original-pull-secret") pod "global-pull-secret-syncer-bgpx8" (UID: "6aecdeeb-4efb-413e-a26c-9270468062a8") : object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:41.507471 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:41.507448    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 00:04:41.507471 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:41.507474    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 00:04:41.507634 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:41.507488    2574 projected.go:196] Error preparing data for projected volume kube-api-access-k7dvp for pod openshift-network-diagnostics/network-check-target-ttc95: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:41.507634 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:41.507535    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d683af01-29f3-4634-ae6d-3de8a5c4fd85-kube-api-access-k7dvp podName:d683af01-29f3-4634-ae6d-3de8a5c4fd85 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:45.507520329 +0000 UTC m=+10.265292904 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7dvp" (UniqueName: "kubernetes.io/projected/d683af01-29f3-4634-ae6d-3de8a5c4fd85-kube-api-access-k7dvp") pod "network-check-target-ttc95" (UID: "d683af01-29f3-4634-ae6d-3de8a5c4fd85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:41.507634 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:41.507604    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:41.507799 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:41.507639    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs podName:a173095c-7c94-4533-8529-81b464886f05 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:45.507627904 +0000 UTC m=+10.265400476 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs") pod "network-metrics-daemon-phwfk" (UID: "a173095c-7c94-4533-8529-81b464886f05") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:41.793782 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:41.793079    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:04:41.793782 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:41.793217    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phwfk" podUID="a173095c-7c94-4533-8529-81b464886f05"
Nov 03 00:04:41.793782 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:41.793603    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:04:41.793782 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:41.793693    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttc95" podUID="d683af01-29f3-4634-ae6d-3de8a5c4fd85"
Nov 03 00:04:42.792539 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:42.792002    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:42.792539 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:42.792143    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgpx8" podUID="6aecdeeb-4efb-413e-a26c-9270468062a8"
Nov 03 00:04:42.912650 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:42.911991    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ld685" event={"ID":"47258bdf-947a-4a28-86f4-02871f64941a","Type":"ContainerStarted","Data":"3168e33490509d0bb7e5bdda7540ba25de13ca526d148465955cdc6f529c8410"}
Nov 03 00:04:43.502430 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:43.502220    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ld685"
Nov 03 00:04:43.502887 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:43.502864    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ld685"
Nov 03 00:04:43.549670 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:43.549608    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ld685" podStartSLOduration=3.539909694 podStartE2EDuration="7.549591379s" totalImagesPullingTime="4.009681685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:42.949008648 +0000 UTC m=+7.706781224" watchObservedRunningTime="2026-11-03 00:04:43.549591379 +0000 UTC m=+8.307363954"
Nov 03 00:04:43.794673 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:43.794030    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:04:43.794673 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:43.794128    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phwfk" podUID="a173095c-7c94-4533-8529-81b464886f05"
Nov 03 00:04:43.794673 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:43.794511    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:04:43.794673 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:43.794604    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttc95" podUID="d683af01-29f3-4634-ae6d-3de8a5c4fd85"
Nov 03 00:04:43.921857 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:43.921211    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rtkhb" event={"ID":"ffb2a03c-414c-4d68-ab86-212c554270e2","Type":"ContainerStarted","Data":"3987d7658a7b48321bee115bdfedb57b692ef834c2ad2278d22108a7fb88c58f"}
Nov 03 00:04:43.928689 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:43.928635    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4" event={"ID":"61d5adce-3738-4cad-aace-39d9d33a739a","Type":"ContainerStarted","Data":"38e432e2afc704f75abf8d1e4a1a385249604edd7a1c3ce00883fc02a81f50ec"}
Nov 03 00:04:43.957093 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:43.957032    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rtkhb" podStartSLOduration=2.988983822 podStartE2EDuration="7.957015643s" totalImagesPullingTime="4.968031821s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:43.956506316 +0000 UTC m=+8.714278890" watchObservedRunningTime="2026-11-03 00:04:43.957015643 +0000 UTC m=+8.714788218"
Nov 03 00:04:44.793256 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:44.792666    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:44.793256 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:44.792807    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgpx8" podUID="6aecdeeb-4efb-413e-a26c-9270468062a8"
Nov 03 00:04:44.931145 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:44.931117    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 00:04:45.325729 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:45.325707    2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 00:04:45.544041 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:45.543999    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6aecdeeb-4efb-413e-a26c-9270468062a8-original-pull-secret\") pod \"global-pull-secret-syncer-bgpx8\" (UID: \"6aecdeeb-4efb-413e-a26c-9270468062a8\") " pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:45.544214 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:45.544073    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7dvp\" (UniqueName: \"kubernetes.io/projected/d683af01-29f3-4634-ae6d-3de8a5c4fd85-kube-api-access-k7dvp\") pod \"network-check-target-ttc95\" (UID: \"d683af01-29f3-4634-ae6d-3de8a5c4fd85\") " pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:04:45.544214 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:45.544112    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs\") pod \"network-metrics-daemon-phwfk\" (UID: \"a173095c-7c94-4533-8529-81b464886f05\") " pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:04:45.544323 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:45.544237    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:45.544323 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:45.544299    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs podName:a173095c-7c94-4533-8529-81b464886f05 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:53.544280249 +0000 UTC m=+18.302052820 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs") pod "network-metrics-daemon-phwfk" (UID: "a173095c-7c94-4533-8529-81b464886f05") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:45.544743 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:45.544725    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:45.544799 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:45.544784    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aecdeeb-4efb-413e-a26c-9270468062a8-original-pull-secret podName:6aecdeeb-4efb-413e-a26c-9270468062a8 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:53.544768729 +0000 UTC m=+18.302541298 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6aecdeeb-4efb-413e-a26c-9270468062a8-original-pull-secret") pod "global-pull-secret-syncer-bgpx8" (UID: "6aecdeeb-4efb-413e-a26c-9270468062a8") : object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:45.544876 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:45.544859    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 00:04:45.544876 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:45.544879    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 00:04:45.545034 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:45.544890    2574 projected.go:196] Error preparing data for projected volume kube-api-access-k7dvp for pod openshift-network-diagnostics/network-check-target-ttc95: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:45.545034 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:45.544941    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d683af01-29f3-4634-ae6d-3de8a5c4fd85-kube-api-access-k7dvp podName:d683af01-29f3-4634-ae6d-3de8a5c4fd85 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:53.544929467 +0000 UTC m=+18.302702032 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7dvp" (UniqueName: "kubernetes.io/projected/d683af01-29f3-4634-ae6d-3de8a5c4fd85-kube-api-access-k7dvp") pod "network-check-target-ttc95" (UID: "d683af01-29f3-4634-ae6d-3de8a5c4fd85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:45.746589 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:45.746314    2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T00:04:45.325725094Z","UUID":"1108d8ba-918b-445e-96c7-788818c5700b","Handler":null,"Name":"","Endpoint":""}
Nov 03 00:04:45.748249 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:45.748228    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 00:04:45.748352 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:45.748271    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 00:04:45.792656 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:45.792626    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:04:45.792799 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:45.792741    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phwfk" podUID="a173095c-7c94-4533-8529-81b464886f05"
Nov 03 00:04:45.793136 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:45.793116    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:04:45.793260 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:45.793191    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttc95" podUID="d683af01-29f3-4634-ae6d-3de8a5c4fd85"
Nov 03 00:04:45.937192 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:45.937160    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4" event={"ID":"61d5adce-3738-4cad-aace-39d9d33a739a","Type":"ContainerStarted","Data":"5a108d7567ceabd87616446019535ec33f15dfb38a338ab1183f7013830e0735"}
Nov 03 00:04:45.939027 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:45.938995    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bcgzj" event={"ID":"015c1f9d-d4c4-4749-978a-3cd2166c7bf8","Type":"ContainerStarted","Data":"d2cad857b326d2e68d03cd952d8e0da8aca4b1f70c08752049455bb804967969"}
Nov 03 00:04:45.942079 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:45.942048    2574 generic.go:356] "Generic (PLEG): container finished" podID="47ea3616-f54d-4603-a56f-5c549d66eb81" containerID="05e5ebd187841d1a8498ea1e5ca5bc1d6a9a5c53c454e2b00b84a0f70b5413ed" exitCode=0
Nov 03 00:04:45.942164 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:45.942087    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bk4mx" event={"ID":"47ea3616-f54d-4603-a56f-5c549d66eb81","Type":"ContainerDied","Data":"05e5ebd187841d1a8498ea1e5ca5bc1d6a9a5c53c454e2b00b84a0f70b5413ed"}
Nov 03 00:04:45.977158 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:45.977111    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-bcgzj" podStartSLOduration=3.606618939 podStartE2EDuration="10.977093991s" totalImagesPullingTime="7.370475052s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:45.975734706 +0000 UTC m=+10.733507280" watchObservedRunningTime="2026-11-03 00:04:45.977093991 +0000 UTC m=+10.734866569"
Nov 03 00:04:46.792160 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:46.792087    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:46.792305 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:46.792211    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgpx8" podUID="6aecdeeb-4efb-413e-a26c-9270468062a8"
Nov 03 00:04:46.945685 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:46.945654    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gtngg" event={"ID":"28e0cef7-ba0d-4546-a193-2f411c441350","Type":"ContainerStarted","Data":"016e492a8dad7a584a32db724ff6bbc3b644e969346e5d700a1d55a0360103eb"}
Nov 03 00:04:46.948386 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:46.948357    2574 generic.go:356] "Generic (PLEG): container finished" podID="47ea3616-f54d-4603-a56f-5c549d66eb81" containerID="23b1c083551963c71dac3fbdd057ee63396e2db102cb3b6e1a2523ce9853d7e6" exitCode=0
Nov 03 00:04:46.948516 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:46.948448    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bk4mx" event={"ID":"47ea3616-f54d-4603-a56f-5c549d66eb81","Type":"ContainerDied","Data":"23b1c083551963c71dac3fbdd057ee63396e2db102cb3b6e1a2523ce9853d7e6"}
Nov 03 00:04:46.950807 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:46.950736    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4" event={"ID":"61d5adce-3738-4cad-aace-39d9d33a739a","Type":"ContainerStarted","Data":"668348f8f021900305336cb9f6cc956b0221bc95344b010161e3eaeba5d170de"}
Nov 03 00:04:46.985165 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:46.985124    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-gtngg" podStartSLOduration=3.209506554 podStartE2EDuration="10.985107933s" totalImagesPullingTime="7.775601379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:46.985076487 +0000 UTC m=+11.742849062" watchObservedRunningTime="2026-11-03 00:04:46.985107933 +0000 UTC m=+11.742880581"
Nov 03 00:04:47.065667 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:47.065589    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4" podStartSLOduration=4.091087305 podStartE2EDuration="12.065570563s" totalImagesPullingTime="7.974483258s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:47.06496137 +0000 UTC m=+11.822733944" watchObservedRunningTime="2026-11-03 00:04:47.065570563 +0000 UTC m=+11.823343186"
Nov 03 00:04:47.792357 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:47.792188    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:04:47.792357 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:47.792315    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phwfk" podUID="a173095c-7c94-4533-8529-81b464886f05"
Nov 03 00:04:47.792716 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:47.792590    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:04:47.792716 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:47.792676    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttc95" podUID="d683af01-29f3-4634-ae6d-3de8a5c4fd85"
Nov 03 00:04:47.955503 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:47.955466    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7fxg5" event={"ID":"7c819325-a41e-445e-858d-fa52911383fe","Type":"ContainerStarted","Data":"b88a43b2128335ef2f64a0542314129cc994e851a38fd07ad6db26defd5e7ab1"}
Nov 03 00:04:47.958351 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:47.958319    2574 generic.go:356] "Generic (PLEG): container finished" podID="47ea3616-f54d-4603-a56f-5c549d66eb81" containerID="6699297e5cbee148f2ce124ae59f66d58d93aeb4374a6dfbf76dd2760e3175f1" exitCode=0
Nov 03 00:04:47.958471 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:47.958402    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bk4mx" event={"ID":"47ea3616-f54d-4603-a56f-5c549d66eb81","Type":"ContainerDied","Data":"6699297e5cbee148f2ce124ae59f66d58d93aeb4374a6dfbf76dd2760e3175f1"}
Nov 03 00:04:47.959832 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:47.959797    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f4p5q" event={"ID":"908118ed-04c5-483d-bb93-61b5d1116259","Type":"ContainerStarted","Data":"d57da5718f89a814a06ebbf26a5ee10e1fea8ae93692536420526e55824dbc82"}
Nov 03 00:04:47.992264 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:47.992215    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7fxg5" podStartSLOduration=2.334252967 podStartE2EDuration="11.992199741s" totalImagesPullingTime="9.657946774s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:47.991983459 +0000 UTC m=+12.749756035" watchObservedRunningTime="2026-11-03 00:04:47.992199741 +0000 UTC m=+12.749972316"
Nov 03 00:04:48.067257 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:48.067181    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f4p5q" podStartSLOduration=4.423376263 podStartE2EDuration="12.067169215s" totalImagesPullingTime="7.643792952s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:48.066849535 +0000 UTC m=+12.824622109" watchObservedRunningTime="2026-11-03 00:04:48.067169215 +0000 UTC m=+12.824941788"
Nov 03 00:04:48.792629 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:48.792597    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:48.792817 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:48.792708    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgpx8" podUID="6aecdeeb-4efb-413e-a26c-9270468062a8"
Nov 03 00:04:49.792635 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:49.792464    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:04:49.793027 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:49.792464    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:04:49.793027 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:49.792741    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttc95" podUID="d683af01-29f3-4634-ae6d-3de8a5c4fd85"
Nov 03 00:04:49.793027 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:49.792863    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phwfk" podUID="a173095c-7c94-4533-8529-81b464886f05"
Nov 03 00:04:49.969565 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:49.969526    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bk4mx" event={"ID":"47ea3616-f54d-4603-a56f-5c549d66eb81","Type":"ContainerStarted","Data":"417d4b66db938b225546c3b92659b22f8bc814d10295a67151539df47db5e44f"}
Nov 03 00:04:50.792505 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:50.792470    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:50.792711 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:50.792596    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgpx8" podUID="6aecdeeb-4efb-413e-a26c-9270468062a8"
Nov 03 00:04:50.974302 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:50.974196    2574 generic.go:356] "Generic (PLEG): container finished" podID="47ea3616-f54d-4603-a56f-5c549d66eb81" containerID="417d4b66db938b225546c3b92659b22f8bc814d10295a67151539df47db5e44f" exitCode=0
Nov 03 00:04:50.974302 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:50.974249    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bk4mx" event={"ID":"47ea3616-f54d-4603-a56f-5c549d66eb81","Type":"ContainerDied","Data":"417d4b66db938b225546c3b92659b22f8bc814d10295a67151539df47db5e44f"}
Nov 03 00:04:51.792394 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:51.792360    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:04:51.792540 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:51.792494    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phwfk" podUID="a173095c-7c94-4533-8529-81b464886f05"
Nov 03 00:04:51.792761 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:51.792733    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:04:51.793168 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:51.792821    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttc95" podUID="d683af01-29f3-4634-ae6d-3de8a5c4fd85"
Nov 03 00:04:51.978553 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:51.978525    2574 generic.go:356] "Generic (PLEG): container finished" podID="47ea3616-f54d-4603-a56f-5c549d66eb81" containerID="a89a20f6bbb90edb384e916fc2954cf3298fa440d2f6002e1d08c10a14bb3299" exitCode=0
Nov 03 00:04:51.978684 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:51.978561    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bk4mx" event={"ID":"47ea3616-f54d-4603-a56f-5c549d66eb81","Type":"ContainerDied","Data":"a89a20f6bbb90edb384e916fc2954cf3298fa440d2f6002e1d08c10a14bb3299"}
Nov 03 00:04:52.792212 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:52.792011    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:52.792349 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:52.792326    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgpx8" podUID="6aecdeeb-4efb-413e-a26c-9270468062a8"
Nov 03 00:04:52.983409 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:52.983350    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:04:52.983771 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:52.983651    2574 generic.go:356] "Generic (PLEG): container finished" podID="6cb79353-f35d-4dde-ad87-f0d03352cc49" containerID="e38570332cda148b0544675c919794c68ea944b32e69790d0a08eb4daa730a6f" exitCode=1
Nov 03 00:04:52.983771 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:52.983733    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr" event={"ID":"6cb79353-f35d-4dde-ad87-f0d03352cc49","Type":"ContainerStarted","Data":"0099c018c0664e9536220d690493be721d3ad40bd30cc383be2f8eca7cc0ddd7"}
Nov 03 00:04:52.983889 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:52.983815    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr" event={"ID":"6cb79353-f35d-4dde-ad87-f0d03352cc49","Type":"ContainerStarted","Data":"31ca7b9b9a1f7f396de1ec01e1fd8b2284aa91580f33f0d6deac4068886b903b"}
Nov 03 00:04:52.983889 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:52.983866    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr" event={"ID":"6cb79353-f35d-4dde-ad87-f0d03352cc49","Type":"ContainerStarted","Data":"77c3b60f0a5102c22f58216638e6bc07982a3e552f65885d22eda6adf89f39a7"}
Nov 03 00:04:52.984012 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:52.983933    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr" event={"ID":"6cb79353-f35d-4dde-ad87-f0d03352cc49","Type":"ContainerStarted","Data":"36c715606a22c963c23f7eac1a33b50d94aa2b03e1fe4329ec1d654cba2b7097"}
Nov 03 00:04:52.984012 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:52.983982    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr" event={"ID":"6cb79353-f35d-4dde-ad87-f0d03352cc49","Type":"ContainerDied","Data":"e38570332cda148b0544675c919794c68ea944b32e69790d0a08eb4daa730a6f"}
Nov 03 00:04:52.984101 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:52.984032    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr" event={"ID":"6cb79353-f35d-4dde-ad87-f0d03352cc49","Type":"ContainerStarted","Data":"f19de77db270e86c5b7f0e6268792b22537bd21e4f6f57ac4e22fdb1ac2daeb1"}
Nov 03 00:04:52.986945 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:52.986919    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bk4mx" event={"ID":"47ea3616-f54d-4603-a56f-5c549d66eb81","Type":"ContainerStarted","Data":"33f9e15d4ad7ea38fbbf05c8bfd3c8f8759aecb6323fdf056c5d487e48ec50d3"}
Nov 03 00:04:53.027410 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:53.027374    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bk4mx" podStartSLOduration=7.974209544 podStartE2EDuration="18.027363456s" totalImagesPullingTime="10.053153912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:53.027008648 +0000 UTC m=+17.784781222" watchObservedRunningTime="2026-11-03 00:04:53.027363456 +0000 UTC m=+17.785136024"
Nov 03 00:04:53.600818 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:53.600789    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6aecdeeb-4efb-413e-a26c-9270468062a8-original-pull-secret\") pod \"global-pull-secret-syncer-bgpx8\" (UID: \"6aecdeeb-4efb-413e-a26c-9270468062a8\") " pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:53.600960 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:53.600827    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7dvp\" (UniqueName: \"kubernetes.io/projected/d683af01-29f3-4634-ae6d-3de8a5c4fd85-kube-api-access-k7dvp\") pod \"network-check-target-ttc95\" (UID: \"d683af01-29f3-4634-ae6d-3de8a5c4fd85\") " pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:04:53.600960 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:53.600845    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs\") pod \"network-metrics-daemon-phwfk\" (UID: \"a173095c-7c94-4533-8529-81b464886f05\") " pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:04:53.600960 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:53.600934    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:53.600960 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:53.600946    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 00:04:53.601094 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:53.600964    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 00:04:53.601094 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:53.600973    2574 projected.go:196] Error preparing data for projected volume kube-api-access-k7dvp for pod openshift-network-diagnostics/network-check-target-ttc95: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:53.601094 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:53.600980    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs podName:a173095c-7c94-4533-8529-81b464886f05 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:09.600968054 +0000 UTC m=+34.358740619 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs") pod "network-metrics-daemon-phwfk" (UID: "a173095c-7c94-4533-8529-81b464886f05") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:53.601094 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:53.600934    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:53.601094 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:53.601010    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d683af01-29f3-4634-ae6d-3de8a5c4fd85-kube-api-access-k7dvp podName:d683af01-29f3-4634-ae6d-3de8a5c4fd85 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:09.600999863 +0000 UTC m=+34.358772414 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7dvp" (UniqueName: "kubernetes.io/projected/d683af01-29f3-4634-ae6d-3de8a5c4fd85-kube-api-access-k7dvp") pod "network-check-target-ttc95" (UID: "d683af01-29f3-4634-ae6d-3de8a5c4fd85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:53.601094 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:53.601055    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aecdeeb-4efb-413e-a26c-9270468062a8-original-pull-secret podName:6aecdeeb-4efb-413e-a26c-9270468062a8 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:09.601043 +0000 UTC m=+34.358815552 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6aecdeeb-4efb-413e-a26c-9270468062a8-original-pull-secret") pod "global-pull-secret-syncer-bgpx8" (UID: "6aecdeeb-4efb-413e-a26c-9270468062a8") : object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:53.792193 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:53.792167    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:04:53.792330 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:53.792291    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phwfk" podUID="a173095c-7c94-4533-8529-81b464886f05"
Nov 03 00:04:53.792409 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:53.792330    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:04:53.792458 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:53.792425    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttc95" podUID="d683af01-29f3-4634-ae6d-3de8a5c4fd85"
Nov 03 00:04:54.792947 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:54.792926    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:54.793242 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:54.793079    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgpx8" podUID="6aecdeeb-4efb-413e-a26c-9270468062a8"
Nov 03 00:04:54.994001 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:54.993974    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:04:54.994333 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:54.994315    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr" event={"ID":"6cb79353-f35d-4dde-ad87-f0d03352cc49","Type":"ContainerStarted","Data":"392b3fbc31cef1799495cddc04fdc603ace76ddb577f745f5f4b8536e2a9ac52"}
Nov 03 00:04:55.793314 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:55.793288    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:04:55.793778 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:55.793361    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttc95" podUID="d683af01-29f3-4634-ae6d-3de8a5c4fd85"
Nov 03 00:04:55.793778 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:55.793456    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:04:55.793778 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:55.793536    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phwfk" podUID="a173095c-7c94-4533-8529-81b464886f05"
Nov 03 00:04:56.792119 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:56.792077    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:56.792309 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:56.792197    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgpx8" podUID="6aecdeeb-4efb-413e-a26c-9270468062a8"
Nov 03 00:04:57.002866 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:57.002843    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:04:57.003380 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:57.003178    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr" event={"ID":"6cb79353-f35d-4dde-ad87-f0d03352cc49","Type":"ContainerStarted","Data":"b3a5e36748e8b50acec09829bb9e42eda07c850b6bac29b039065cd7631d8a30"}
Nov 03 00:04:57.003496 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:57.003471    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:57.003613 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:57.003549    2574 scope.go:122] "RemoveContainer" containerID="e38570332cda148b0544675c919794c68ea944b32e69790d0a08eb4daa730a6f"
Nov 03 00:04:57.003666 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:57.003553    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:57.019204 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:57.019184    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:57.021381 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:57.021362    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:04:57.792762 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:57.792510    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:04:57.792943 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:57.792564    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:04:57.792943 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:57.792873    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phwfk" podUID="a173095c-7c94-4533-8529-81b464886f05"
Nov 03 00:04:57.792943 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:57.792895    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttc95" podUID="d683af01-29f3-4634-ae6d-3de8a5c4fd85"
Nov 03 00:04:58.008425 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:58.008402    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:04:58.008792 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:58.008770    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr" event={"ID":"6cb79353-f35d-4dde-ad87-f0d03352cc49","Type":"ContainerStarted","Data":"5b1eea1eac986aeb250070edd56df1789299a8e8ef8a64531ba63497344617d7"}
Nov 03 00:04:58.008864 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:58.008850    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 00:04:58.087375 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:58.087285    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr" podStartSLOduration=8.784049042 podStartE2EDuration="23.087269876s" totalImagesPullingTime="14.303220834s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:58.086900456 +0000 UTC m=+22.844673030" watchObservedRunningTime="2026-11-03 00:04:58.087269876 +0000 UTC m=+22.845042451"
Nov 03 00:04:58.792005 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:58.791981    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:58.792148 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:58.792071    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgpx8" podUID="6aecdeeb-4efb-413e-a26c-9270468062a8"
Nov 03 00:04:58.929654 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:58.929618    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ttc95"]
Nov 03 00:04:58.929811 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:58.929714    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:04:58.929811 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:58.929797    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttc95" podUID="d683af01-29f3-4634-ae6d-3de8a5c4fd85"
Nov 03 00:04:58.932655 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:58.932632    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bgpx8"]
Nov 03 00:04:58.940743 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:58.940725    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-phwfk"]
Nov 03 00:04:58.940934 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:58.940803    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:04:58.940934 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:58.940884    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phwfk" podUID="a173095c-7c94-4533-8529-81b464886f05"
Nov 03 00:04:59.010652 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:59.010627    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:04:59.010978 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:04:59.010705    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 00:04:59.010978 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:04:59.010733    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgpx8" podUID="6aecdeeb-4efb-413e-a26c-9270468062a8"
Nov 03 00:05:00.344260 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:00.344221    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:00.792926 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:00.792891    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:05:00.793091 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:00.792949    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:05:00.793091 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:00.792968    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:05:00.793091 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:00.793045    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgpx8" podUID="6aecdeeb-4efb-413e-a26c-9270468062a8"
Nov 03 00:05:00.793196 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:00.793162    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phwfk" podUID="a173095c-7c94-4533-8529-81b464886f05"
Nov 03 00:05:00.793235 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:00.793223    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttc95" podUID="d683af01-29f3-4634-ae6d-3de8a5c4fd85"
Nov 03 00:05:01.532750 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:01.532715    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ld685"
Nov 03 00:05:01.533166 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:01.532864    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 00:05:01.533455 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:01.533434    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ld685"
Nov 03 00:05:02.792988 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:02.792958    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:05:02.792988 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:02.792982    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:05:02.793486 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:02.793053    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:05:02.793486 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:02.793069    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttc95" podUID="d683af01-29f3-4634-ae6d-3de8a5c4fd85"
Nov 03 00:05:02.793486 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:02.793139    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phwfk" podUID="a173095c-7c94-4533-8529-81b464886f05"
Nov 03 00:05:02.793486 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:02.793221    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgpx8" podUID="6aecdeeb-4efb-413e-a26c-9270468062a8"
Nov 03 00:05:04.793101 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:04.793057    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:05:04.793101 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:04.793090    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:05:04.793624 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:04.793076    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:05:04.793624 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:04.793191    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-phwfk" podUID="a173095c-7c94-4533-8529-81b464886f05"
Nov 03 00:05:04.793624 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:04.793263    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgpx8" podUID="6aecdeeb-4efb-413e-a26c-9270468062a8"
Nov 03 00:05:04.793624 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:04.793316    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttc95" podUID="d683af01-29f3-4634-ae6d-3de8a5c4fd85"
Nov 03 00:05:06.063526 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.063444    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeReady"
Nov 03 00:05:06.064005 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.063609    2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 00:05:06.112874 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.112839    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2mp9l"]
Nov 03 00:05:06.159701 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.159662    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"]
Nov 03 00:05:06.160050 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.159847    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-2mp9l"
Nov 03 00:05:06.175624 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.175597    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lt4f5"]
Nov 03 00:05:06.175840 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.175818    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"
Nov 03 00:05:06.180823 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.180796    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 03 00:05:06.180983 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.180860    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 03 00:05:06.182518 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.182490    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7564dc7d49-qtdkm"]
Nov 03 00:05:06.183539 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.183446    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-lt4f5"
Nov 03 00:05:06.199490 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.199375    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7df847f57-khf2f"]
Nov 03 00:05:06.199573 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.199494    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:06.200955 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.200933    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 03 00:05:06.215095 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.215061    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc"]
Nov 03 00:05:06.215282 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.215192    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:06.229186 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.229155    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29894400-s5xp5"]
Nov 03 00:05:06.229342 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.229296    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc"
Nov 03 00:05:06.241172 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.241144    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lnk6s"]
Nov 03 00:05:06.241331 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.241260    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29894400-s5xp5"
Nov 03 00:05:06.251597 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.251570    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-kzsm6"]
Nov 03 00:05:06.251731 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.251576    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-lnk6s"
Nov 03 00:05:06.255568 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.255529    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5d95d"]
Nov 03 00:05:06.255703 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.255631    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:06.276215 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.276183    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"]
Nov 03 00:05:06.276369 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.276310    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:06.279271 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.279240    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:05:06.279433 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.279286    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 00:05:06.279433 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.279307    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 03 00:05:06.279433 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.279312    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 03 00:05:06.279433 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.279329    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 00:05:06.279433 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.279252    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 00:05:06.279433 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.279252    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 00:05:06.279433 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.279393    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\""
Nov 03 00:05:06.279433 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.279409    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 00:05:06.279433 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.279408    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 03 00:05:06.279433 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.279436    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 00:05:06.279759 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.279449    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 00:05:06.290800 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.290770    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 03 00:05:06.292331 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.292293    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p"]
Nov 03 00:05:06.292503 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.292402    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"
Nov 03 00:05:06.298996 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.298972    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-stats-auth\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:06.299103 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.299008    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-trusted-ca\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:06.299103 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.299030    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jkxn\" (UniqueName: \"kubernetes.io/projected/ee7974f8-92b4-498a-bf54-2c3a353f4b49-kube-api-access-6jkxn\") pod \"kas-connection-checker-685895c7c9-2mp9l\" (UID: \"ee7974f8-92b4-498a-bf54-2c3a353f4b49\") " pod="kube-system/kas-connection-checker-685895c7c9-2mp9l"
Nov 03 00:05:06.299176 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.299105    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kscjb\" (UniqueName: \"kubernetes.io/projected/daa3da9a-3a5f-49a4-8a09-0036fdc5aa44-kube-api-access-kscjb\") pod \"image-pruner-29894400-s5xp5\" (UID: \"daa3da9a-3a5f-49a4-8a09-0036fdc5aa44\") " pod="openshift-image-registry/image-pruner-29894400-s5xp5"
Nov 03 00:05:06.299229 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.299209    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:06.299271 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.299258    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xcfc\" (UniqueName: \"kubernetes.io/projected/12eb002e-9765-4443-b488-6cec02b9c961-kube-api-access-5xcfc\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:06.299304 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.299282    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-image-registry-private-configuration\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:06.299304 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.299299    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-ca-trust-extracted\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:06.299382 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.299314    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38b66660-5608-4e82-a1eb-db3d6ce57ca5-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc\" (UID: \"38b66660-5608-4e82-a1eb-db3d6ce57ca5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc"
Nov 03 00:05:06.299382 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.299365    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/daa3da9a-3a5f-49a4-8a09-0036fdc5aa44-serviceca\") pod \"image-pruner-29894400-s5xp5\" (UID: \"daa3da9a-3a5f-49a4-8a09-0036fdc5aa44\") " pod="openshift-image-registry/image-pruner-29894400-s5xp5"
Nov 03 00:05:06.299497 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.299476    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-default-certificate\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:06.299541 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.299515    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffkwv\" (UID: \"dbea6432-5ba3-49f3-8a83-db2605a3aa2c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"
Nov 03 00:05:06.299591 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.299543    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-certificates\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:06.299647 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.299594    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2r5h\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-kube-api-access-s2r5h\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:06.299707 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.299689    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:06.299763 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.299726    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ffkwv\" (UID: \"dbea6432-5ba3-49f3-8a83-db2605a3aa2c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"
Nov 03 00:05:06.299818 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.299759    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-installation-pull-secrets\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:06.299818 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.299790    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38b66660-5608-4e82-a1eb-db3d6ce57ca5-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc\" (UID: \"38b66660-5608-4e82-a1eb-db3d6ce57ca5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc"
Nov 03 00:05:06.299893 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.299824    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wm66\" (UniqueName: \"kubernetes.io/projected/913d1e5a-3dfc-4282-b5f0-95c4484b31da-kube-api-access-2wm66\") pod \"kas-connection-checker-685895c7c9-lt4f5\" (UID: \"913d1e5a-3dfc-4282-b5f0-95c4484b31da\") " pod="kube-system/kas-connection-checker-685895c7c9-lt4f5"
Nov 03 00:05:06.299893 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.299854    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:06.299893 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.299875    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-bound-sa-token\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:06.300013 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.299902    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chv2w\" (UniqueName: \"kubernetes.io/projected/38b66660-5608-4e82-a1eb-db3d6ce57ca5-kube-api-access-chv2w\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc\" (UID: \"38b66660-5608-4e82-a1eb-db3d6ce57ca5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc"
Nov 03 00:05:06.300013 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.299986    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfdhj\" (UniqueName: \"kubernetes.io/projected/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-kube-api-access-cfdhj\") pod \"cluster-monitoring-operator-884576c57-ffkwv\" (UID: \"dbea6432-5ba3-49f3-8a83-db2605a3aa2c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"
Nov 03 00:05:06.300306 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.300288    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 00:05:06.302634 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.302609    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 00:05:06.302738 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.302636    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 00:05:06.302774 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.302747    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-hcj5z\""
Nov 03 00:05:06.302918 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.302880    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 03 00:05:06.302918 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.302891    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rns5k\""
Nov 03 00:05:06.303097 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.302941    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 03 00:05:06.303097 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.302951    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 03 00:05:06.303097 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.302977    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 03 00:05:06.303097 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.302992    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 03 00:05:06.303097 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.303001    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 00:05:06.303097 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.303018    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 03 00:05:06.303097 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.303033    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s7x9g\""
Nov 03 00:05:06.303097 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.303040    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-rbmkp\""
Nov 03 00:05:06.303097 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.303067    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-vm5wz\""
Nov 03 00:05:06.303489 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.303162    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 00:05:06.303489 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.303416    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v4sr6"]
Nov 03 00:05:06.303551 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.303531    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p"
Nov 03 00:05:06.305995 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.305939    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 03 00:05:06.306113 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.306086    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-wvfrg\""
Nov 03 00:05:06.311775 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.311751    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-8mjkf"]
Nov 03 00:05:06.311930 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.311897    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v4sr6"
Nov 03 00:05:06.316788 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.316736    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 03 00:05:06.325861 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.325837    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-tbknj\""
Nov 03 00:05:06.340468 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.340435    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2m6bp"]
Nov 03 00:05:06.340646 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.340520    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8mjkf"
Nov 03 00:05:06.354427 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.354383    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-668bbfbdd4-9m8pn"]
Nov 03 00:05:06.354610 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.354520    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2m6bp"
Nov 03 00:05:06.370211 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.370177    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2mp9l"]
Nov 03 00:05:06.370211 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.370223    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-n68xd"]
Nov 03 00:05:06.370683 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.370319    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:06.377501 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.377473    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lt4f5"]
Nov 03 00:05:06.377627 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.377508    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"]
Nov 03 00:05:06.377627 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.377526    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lcdlg"]
Nov 03 00:05:06.377627 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.377591    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n68xd"
Nov 03 00:05:06.379337 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.379303    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 03 00:05:06.379555 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.379538    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:05:06.379976 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.379958    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 03 00:05:06.380897 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.380874    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 03 00:05:06.380897 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.380892    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:05:06.381132 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.381112    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 03 00:05:06.381200 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.381174    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 03 00:05:06.381267 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.381205    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:05:06.381352 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.381334    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 03 00:05:06.381593 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.381574    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 00:05:06.381736 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.381720    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:05:06.381798 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.381739    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 03 00:05:06.381798 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.381769    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 00:05:06.381981 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.381968    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 03 00:05:06.383989 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.383969    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 03 00:05:06.391597 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.391577    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 00:05:06.392325 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.392302    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 00:05:06.392584 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.392566    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 03 00:05:06.392660 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.392575    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ct6nh\""
Nov 03 00:05:06.392660 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.392620    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-shzgc\""
Nov 03 00:05:06.394584 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.394568    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 00:05:06.400640 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.400604    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc"]
Nov 03 00:05:06.400781 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.400707    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:06.400781 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.400729    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29894400-s5xp5"]
Nov 03 00:05:06.400781 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.400752    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"]
Nov 03 00:05:06.400781 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.400777    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-kzsm6"]
Nov 03 00:05:06.400967 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.400798    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v4sr6"]
Nov 03 00:05:06.400967 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.400818    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2m6bp"]
Nov 03 00:05:06.400967 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.400824    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:06.400967 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.400839    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-8mjkf"]
Nov 03 00:05:06.400967 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.400849    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-bound-sa-token\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:06.400967 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.400859    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p"]
Nov 03 00:05:06.400967 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.400879    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7564dc7d49-qtdkm"]
Nov 03 00:05:06.400967 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.400879    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9rtk\" (UniqueName: \"kubernetes.io/projected/667223ca-ffbc-4182-beb6-67e68116171f-kube-api-access-t9rtk\") pod \"network-check-source-6f78d59f99-8mjkf\" (UID: \"667223ca-ffbc-4182-beb6-67e68116171f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8mjkf"
Nov 03 00:05:06.400967 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.400899    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7df847f57-khf2f"]
Nov 03 00:05:06.400967 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.400937    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lnk6s"]
Nov 03 00:05:06.400967 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.400960    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-668bbfbdd4-9m8pn"]
Nov 03 00:05:06.401427 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.400978    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lcdlg"]
Nov 03 00:05:06.401427 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.401001    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n68xd"]
Nov 03 00:05:06.401427 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.401019    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5d95d"]
Nov 03 00:05:06.401427 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:06.401020    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 00:05:06.401427 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.400933    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffkwv\" (UID: \"dbea6432-5ba3-49f3-8a83-db2605a3aa2c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"
Nov 03 00:05:06.401427 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:06.401091    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls podName:dbea6432-5ba3-49f3-8a83-db2605a3aa2c nodeName:}" failed. No retries permitted until 2026-11-03 00:05:06.901072144 +0000 UTC m=+31.658844722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ffkwv" (UID: "dbea6432-5ba3-49f3-8a83-db2605a3aa2c") : secret "cluster-monitoring-operator-tls" not found
Nov 03 00:05:06.401427 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.401109    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-certificates\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:06.401427 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:06.401137    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 00:05:06.401427 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.401135    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-tmp\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:06.401427 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:06.401177    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs podName:12eb002e-9765-4443-b488-6cec02b9c961 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:06.901165062 +0000 UTC m=+31.658937617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs") pod "router-default-7df847f57-khf2f" (UID: "12eb002e-9765-4443-b488-6cec02b9c961") : secret "router-metrics-certs-default" not found
Nov 03 00:05:06.401427 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.401213    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-snapshots\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:06.401427 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.401247    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-installation-pull-secrets\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:06.401427 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.401281    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kgwx\" (UniqueName: \"kubernetes.io/projected/d8606abd-d458-4bee-b150-461becfbb83b-kube-api-access-4kgwx\") pod \"kas-connection-checker-685895c7c9-lnk6s\" (UID: \"d8606abd-d458-4bee-b150-461becfbb83b\") " pod="kube-system/kas-connection-checker-685895c7c9-lnk6s"
Nov 03 00:05:06.401427 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.401311    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr45n\" (UniqueName: \"kubernetes.io/projected/eddf03ef-53ea-4fc0-9f3f-da7d741cf06a-kube-api-access-wr45n\") pod \"volume-data-source-validator-58c9f5f46d-v4sr6\" (UID: \"eddf03ef-53ea-4fc0-9f3f-da7d741cf06a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v4sr6"
Nov 03 00:05:06.401427 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.401353    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wm66\" (UniqueName: \"kubernetes.io/projected/913d1e5a-3dfc-4282-b5f0-95c4484b31da-kube-api-access-2wm66\") pod \"kas-connection-checker-685895c7c9-lt4f5\" (UID: \"913d1e5a-3dfc-4282-b5f0-95c4484b31da\") " pod="kube-system/kas-connection-checker-685895c7c9-lt4f5"
Nov 03 00:05:06.401427 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.401378    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:06.402151 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.401453    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chv2w\" (UniqueName: \"kubernetes.io/projected/38b66660-5608-4e82-a1eb-db3d6ce57ca5-kube-api-access-chv2w\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc\" (UID: \"38b66660-5608-4e82-a1eb-db3d6ce57ca5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc"
Nov 03 00:05:06.402151 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.401484    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-serving-cert\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:06.402151 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:06.401499    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 00:05:06.402151 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:06.401513    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7564dc7d49-qtdkm: secret "image-registry-tls" not found
Nov 03 00:05:06.402151 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.401533    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-trusted-ca\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:06.402151 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:06.401563    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls podName:ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:06.901549075 +0000 UTC m=+31.659321641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls") pod "image-registry-7564dc7d49-qtdkm" (UID: "ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3") : secret "image-registry-tls" not found
Nov 03 00:05:06.402151 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.401582    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnl7p\" (UniqueName: \"kubernetes.io/projected/a8dde726-eabf-405e-9d38-a110274353ca-kube-api-access-jnl7p\") pod \"cluster-samples-operator-6ddb6c8b94-hcrh6\" (UID: \"a8dde726-eabf-405e-9d38-a110274353ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"
Nov 03 00:05:06.402151 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.401638    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/689b30e8-936e-49a4-a775-4ce908ad1ab8-config\") pod \"service-ca-operator-544d749c8c-w6k5p\" (UID: \"689b30e8-936e-49a4-a775-4ce908ad1ab8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p"
Nov 03 00:05:06.402151 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.401667    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-hcrh6\" (UID: \"a8dde726-eabf-405e-9d38-a110274353ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"
Nov 03 00:05:06.402151 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.401696    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:06.402151 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.401723    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-image-registry-private-configuration\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:06.402151 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.401752    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38b66660-5608-4e82-a1eb-db3d6ce57ca5-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc\" (UID: \"38b66660-5608-4e82-a1eb-db3d6ce57ca5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc"
Nov 03 00:05:06.402151 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.401794    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xcfc\" (UniqueName: \"kubernetes.io/projected/12eb002e-9765-4443-b488-6cec02b9c961-kube-api-access-5xcfc\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:06.402151 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.401835    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-certificates\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:06.402151 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.401888    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c1de1047-c4c9-4d0b-8f72-bafc79987c2e-trusted-ca\") pod \"console-operator-7cd56b8dbc-5d95d\" (UID: \"c1de1047-c4c9-4d0b-8f72-bafc79987c2e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:06.402821 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:06.401903    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle podName:12eb002e-9765-4443-b488-6cec02b9c961 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:06.901891425 +0000 UTC m=+31.659664005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle") pod "router-default-7df847f57-khf2f" (UID: "12eb002e-9765-4443-b488-6cec02b9c961") : configmap references non-existent config key: service-ca.crt
Nov 03 00:05:06.402821 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.402150    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ffkwv\" (UID: \"dbea6432-5ba3-49f3-8a83-db2605a3aa2c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"
Nov 03 00:05:06.402821 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.402199    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-default-certificate\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:06.402821 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.402229    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2r5h\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-kube-api-access-s2r5h\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:06.402821 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.402258    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:06.402821 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.402289    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38b66660-5608-4e82-a1eb-db3d6ce57ca5-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc\" (UID: \"38b66660-5608-4e82-a1eb-db3d6ce57ca5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc"
Nov 03 00:05:06.402821 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.402317    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw7ww\" (UniqueName: \"kubernetes.io/projected/c1de1047-c4c9-4d0b-8f72-bafc79987c2e-kube-api-access-kw7ww\") pod \"console-operator-7cd56b8dbc-5d95d\" (UID: \"c1de1047-c4c9-4d0b-8f72-bafc79987c2e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:06.402821 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.402350    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1de1047-c4c9-4d0b-8f72-bafc79987c2e-serving-cert\") pod \"console-operator-7cd56b8dbc-5d95d\" (UID: \"c1de1047-c4c9-4d0b-8f72-bafc79987c2e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:06.402821 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.402390    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9pbs\" (UniqueName: \"kubernetes.io/projected/689b30e8-936e-49a4-a775-4ce908ad1ab8-kube-api-access-h9pbs\") pod \"service-ca-operator-544d749c8c-w6k5p\" (UID: \"689b30e8-936e-49a4-a775-4ce908ad1ab8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p"
Nov 03 00:05:06.402821 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.402423    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-service-ca-bundle\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:06.402821 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.402449    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgqk7\" (UniqueName: \"kubernetes.io/projected/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-kube-api-access-rgqk7\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:06.402821 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.402475    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/689b30e8-936e-49a4-a775-4ce908ad1ab8-serving-cert\") pod \"service-ca-operator-544d749c8c-w6k5p\" (UID: \"689b30e8-936e-49a4-a775-4ce908ad1ab8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p"
Nov 03 00:05:06.402821 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.402488    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-trusted-ca\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:06.402821 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.402500    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hcrh6\" (UID: \"a8dde726-eabf-405e-9d38-a110274353ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"
Nov 03 00:05:06.403585 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.402532    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfdhj\" (UniqueName: \"kubernetes.io/projected/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-kube-api-access-cfdhj\") pod \"cluster-monitoring-operator-884576c57-ffkwv\" (UID: \"dbea6432-5ba3-49f3-8a83-db2605a3aa2c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"
Nov 03 00:05:06.403585 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.403083    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38b66660-5608-4e82-a1eb-db3d6ce57ca5-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc\" (UID: \"38b66660-5608-4e82-a1eb-db3d6ce57ca5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc"
Nov 03 00:05:06.403585 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.403114    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-stats-auth\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:06.403585 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.403169    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jkxn\" (UniqueName: \"kubernetes.io/projected/ee7974f8-92b4-498a-bf54-2c3a353f4b49-kube-api-access-6jkxn\") pod \"kas-connection-checker-685895c7c9-2mp9l\" (UID: \"ee7974f8-92b4-498a-bf54-2c3a353f4b49\") " pod="kube-system/kas-connection-checker-685895c7c9-2mp9l"
Nov 03 00:05:06.403585 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.403181    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ffkwv\" (UID: \"dbea6432-5ba3-49f3-8a83-db2605a3aa2c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"
Nov 03 00:05:06.403585 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.403447    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kscjb\" (UniqueName: \"kubernetes.io/projected/daa3da9a-3a5f-49a4-8a09-0036fdc5aa44-kube-api-access-kscjb\") pod \"image-pruner-29894400-s5xp5\" (UID: \"daa3da9a-3a5f-49a4-8a09-0036fdc5aa44\") " pod="openshift-image-registry/image-pruner-29894400-s5xp5"
Nov 03 00:05:06.403888 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.403865    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-ca-trust-extracted\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:06.403974 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.403939    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/daa3da9a-3a5f-49a4-8a09-0036fdc5aa44-serviceca\") pod \"image-pruner-29894400-s5xp5\" (UID: \"daa3da9a-3a5f-49a4-8a09-0036fdc5aa44\") " pod="openshift-image-registry/image-pruner-29894400-s5xp5"
Nov 03 00:05:06.404038 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.403985    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1de1047-c4c9-4d0b-8f72-bafc79987c2e-config\") pod \"console-operator-7cd56b8dbc-5d95d\" (UID: \"c1de1047-c4c9-4d0b-8f72-bafc79987c2e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:06.404400 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.404377    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-ca-trust-extracted\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:06.404940 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.404898    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/daa3da9a-3a5f-49a4-8a09-0036fdc5aa44-serviceca\") pod \"image-pruner-29894400-s5xp5\" (UID: \"daa3da9a-3a5f-49a4-8a09-0036fdc5aa44\") " pod="openshift-image-registry/image-pruner-29894400-s5xp5"
Nov 03 00:05:06.406023 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.405999    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-stats-auth\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:06.406118 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.406000    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-default-certificate\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:06.406118 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.406028    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-installation-pull-secrets\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:06.406118 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.406056    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-image-registry-private-configuration\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:06.406354 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.406335    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38b66660-5608-4e82-a1eb-db3d6ce57ca5-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc\" (UID: \"38b66660-5608-4e82-a1eb-db3d6ce57ca5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc"
Nov 03 00:05:06.422486 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.422217    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chv2w\" (UniqueName: \"kubernetes.io/projected/38b66660-5608-4e82-a1eb-db3d6ce57ca5-kube-api-access-chv2w\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc\" (UID: \"38b66660-5608-4e82-a1eb-db3d6ce57ca5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc"
Nov 03 00:05:06.422848 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.422816    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-bound-sa-token\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:06.423027 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.423006    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xcfc\" (UniqueName: \"kubernetes.io/projected/12eb002e-9765-4443-b488-6cec02b9c961-kube-api-access-5xcfc\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:06.423108 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.423087    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wm66\" (UniqueName: \"kubernetes.io/projected/913d1e5a-3dfc-4282-b5f0-95c4484b31da-kube-api-access-2wm66\") pod \"kas-connection-checker-685895c7c9-lt4f5\" (UID: \"913d1e5a-3dfc-4282-b5f0-95c4484b31da\") " pod="kube-system/kas-connection-checker-685895c7c9-lt4f5"
Nov 03 00:05:06.424490 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.424457    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jkxn\" (UniqueName: \"kubernetes.io/projected/ee7974f8-92b4-498a-bf54-2c3a353f4b49-kube-api-access-6jkxn\") pod \"kas-connection-checker-685895c7c9-2mp9l\" (UID: \"ee7974f8-92b4-498a-bf54-2c3a353f4b49\") " pod="kube-system/kas-connection-checker-685895c7c9-2mp9l"
Nov 03 00:05:06.424490 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.424475    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kscjb\" (UniqueName: \"kubernetes.io/projected/daa3da9a-3a5f-49a4-8a09-0036fdc5aa44-kube-api-access-kscjb\") pod \"image-pruner-29894400-s5xp5\" (UID: \"daa3da9a-3a5f-49a4-8a09-0036fdc5aa44\") " pod="openshift-image-registry/image-pruner-29894400-s5xp5"
Nov 03 00:05:06.424623 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.424582    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfdhj\" (UniqueName: \"kubernetes.io/projected/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-kube-api-access-cfdhj\") pod \"cluster-monitoring-operator-884576c57-ffkwv\" (UID: \"dbea6432-5ba3-49f3-8a83-db2605a3aa2c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"
Nov 03 00:05:06.426952 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.426930    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2r5h\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-kube-api-access-s2r5h\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:06.477516 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.477477    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-2mp9l"
Nov 03 00:05:06.479589 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.479563    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 00:05:06.492797 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.492777    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 00:05:06.492873 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.492802    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7b8cw\""
Nov 03 00:05:06.505235 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.505182    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kgwx\" (UniqueName: \"kubernetes.io/projected/d8606abd-d458-4bee-b150-461becfbb83b-kube-api-access-4kgwx\") pod \"kas-connection-checker-685895c7c9-lnk6s\" (UID: \"d8606abd-d458-4bee-b150-461becfbb83b\") " pod="kube-system/kas-connection-checker-685895c7c9-lnk6s"
Nov 03 00:05:06.505235 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.505228    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr45n\" (UniqueName: \"kubernetes.io/projected/eddf03ef-53ea-4fc0-9f3f-da7d741cf06a-kube-api-access-wr45n\") pod \"volume-data-source-validator-58c9f5f46d-v4sr6\" (UID: \"eddf03ef-53ea-4fc0-9f3f-da7d741cf06a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v4sr6"
Nov 03 00:05:06.505496 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.505256    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbzll\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-kube-api-access-bbzll\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:06.505496 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.505309    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-serving-cert\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:06.505496 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.505352    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-ca-trust-extracted\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:06.505496 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.505429    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bb5193c7-10f4-4436-9c69-20e6087439c2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2m6bp\" (UID: \"bb5193c7-10f4-4436-9c69-20e6087439c2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2m6bp"
Nov 03 00:05:06.505496 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.505477    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnl7p\" (UniqueName: \"kubernetes.io/projected/a8dde726-eabf-405e-9d38-a110274353ca-kube-api-access-jnl7p\") pod \"cluster-samples-operator-6ddb6c8b94-hcrh6\" (UID: \"a8dde726-eabf-405e-9d38-a110274353ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"
Nov 03 00:05:06.505859 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.505578    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/689b30e8-936e-49a4-a775-4ce908ad1ab8-config\") pod \"service-ca-operator-544d749c8c-w6k5p\" (UID: \"689b30e8-936e-49a4-a775-4ce908ad1ab8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p"
Nov 03 00:05:06.505859 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.505625    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-hcrh6\" (UID: \"a8dde726-eabf-405e-9d38-a110274353ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"
Nov 03 00:05:06.505859 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.505752    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-image-registry-private-configuration\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:06.505859 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.505791    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c1de1047-c4c9-4d0b-8f72-bafc79987c2e-trusted-ca\") pod \"console-operator-7cd56b8dbc-5d95d\" (UID: \"c1de1047-c4c9-4d0b-8f72-bafc79987c2e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:06.505859 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.505820    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-bound-sa-token\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:06.506118 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.505877    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:06.506118 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.505905    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw7ww\" (UniqueName: \"kubernetes.io/projected/c1de1047-c4c9-4d0b-8f72-bafc79987c2e-kube-api-access-kw7ww\") pod \"console-operator-7cd56b8dbc-5d95d\" (UID: \"c1de1047-c4c9-4d0b-8f72-bafc79987c2e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:06.506118 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.505962    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-certificates\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:06.506118 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.505993    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkvnm\" (UniqueName: \"kubernetes.io/projected/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-kube-api-access-dkvnm\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:06.506118 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.506025    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/735e8903-5dfe-4cb2-8937-cfc3d09ffa36-cert\") pod \"ingress-canary-n68xd\" (UID: \"735e8903-5dfe-4cb2-8937-cfc3d09ffa36\") " pod="openshift-ingress-canary/ingress-canary-n68xd"
Nov 03 00:05:06.506118 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.506054    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1de1047-c4c9-4d0b-8f72-bafc79987c2e-serving-cert\") pod \"console-operator-7cd56b8dbc-5d95d\" (UID: \"c1de1047-c4c9-4d0b-8f72-bafc79987c2e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:06.506118 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.506078    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:06.506118 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.506108    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-tmp-dir\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:06.506516 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.506142    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9pbs\" (UniqueName: \"kubernetes.io/projected/689b30e8-936e-49a4-a775-4ce908ad1ab8-kube-api-access-h9pbs\") pod \"service-ca-operator-544d749c8c-w6k5p\" (UID: \"689b30e8-936e-49a4-a775-4ce908ad1ab8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p"
Nov 03 00:05:06.506516 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.506164    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-service-ca-bundle\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:06.506516 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.506189    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgqk7\" (UniqueName: \"kubernetes.io/projected/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-kube-api-access-rgqk7\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:06.506516 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.506216    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/689b30e8-936e-49a4-a775-4ce908ad1ab8-serving-cert\") pod \"service-ca-operator-544d749c8c-w6k5p\" (UID: \"689b30e8-936e-49a4-a775-4ce908ad1ab8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p"
Nov 03 00:05:06.506516 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.506244    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hcrh6\" (UID: \"a8dde726-eabf-405e-9d38-a110274353ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"
Nov 03 00:05:06.506516 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.506272    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:06.506516 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.506421    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/689b30e8-936e-49a4-a775-4ce908ad1ab8-config\") pod \"service-ca-operator-544d749c8c-w6k5p\" (UID: \"689b30e8-936e-49a4-a775-4ce908ad1ab8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p"
Nov 03 00:05:06.506516 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.506498    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-hcrh6\" (UID: \"a8dde726-eabf-405e-9d38-a110274353ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"
Nov 03 00:05:06.506876 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:06.506605    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 00:05:06.506876 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:06.506669    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls podName:a8dde726-eabf-405e-9d38-a110274353ca nodeName:}" failed. No retries permitted until 2026-11-03 00:05:07.006650621 +0000 UTC m=+31.764423260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hcrh6" (UID: "a8dde726-eabf-405e-9d38-a110274353ca") : secret "samples-operator-tls" not found
Nov 03 00:05:06.507005 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.506890    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c1de1047-c4c9-4d0b-8f72-bafc79987c2e-trusted-ca\") pod \"console-operator-7cd56b8dbc-5d95d\" (UID: \"c1de1047-c4c9-4d0b-8f72-bafc79987c2e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:06.507005 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.506953    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:06.507109 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.507048    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-config-volume\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:06.507109 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.507083    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6w5f\" (UniqueName: \"kubernetes.io/projected/735e8903-5dfe-4cb2-8937-cfc3d09ffa36-kube-api-access-r6w5f\") pod \"ingress-canary-n68xd\" (UID: \"735e8903-5dfe-4cb2-8937-cfc3d09ffa36\") " pod="openshift-ingress-canary/ingress-canary-n68xd"
Nov 03 00:05:06.507192 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.507104    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-service-ca-bundle\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:06.507192 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.507111    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2m6bp\" (UID: \"bb5193c7-10f4-4436-9c69-20e6087439c2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2m6bp"
Nov 03 00:05:06.507283 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.507217    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1de1047-c4c9-4d0b-8f72-bafc79987c2e-config\") pod \"console-operator-7cd56b8dbc-5d95d\" (UID: \"c1de1047-c4c9-4d0b-8f72-bafc79987c2e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:06.507283 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.507263    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-installation-pull-secrets\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:06.507376 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.507308    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9rtk\" (UniqueName: \"kubernetes.io/projected/667223ca-ffbc-4182-beb6-67e68116171f-kube-api-access-t9rtk\") pod \"network-check-source-6f78d59f99-8mjkf\" (UID: \"667223ca-ffbc-4182-beb6-67e68116171f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8mjkf"
Nov 03 00:05:06.507423 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.507389    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-tmp\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:06.507472 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.507428    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-snapshots\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:06.507472 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.507465    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-trusted-ca\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:06.507770 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.507708    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-tmp\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:06.508245 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.508214    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-snapshots\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:06.508560 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.508541    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-serving-cert\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:06.508987 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.508958    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1de1047-c4c9-4d0b-8f72-bafc79987c2e-config\") pod \"console-operator-7cd56b8dbc-5d95d\" (UID: \"c1de1047-c4c9-4d0b-8f72-bafc79987c2e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:06.509147 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.509130    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/689b30e8-936e-49a4-a775-4ce908ad1ab8-serving-cert\") pod \"service-ca-operator-544d749c8c-w6k5p\" (UID: \"689b30e8-936e-49a4-a775-4ce908ad1ab8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p"
Nov 03 00:05:06.509236 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.509216    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1de1047-c4c9-4d0b-8f72-bafc79987c2e-serving-cert\") pod \"console-operator-7cd56b8dbc-5d95d\" (UID: \"c1de1047-c4c9-4d0b-8f72-bafc79987c2e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:06.524087 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.523954    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-lt4f5"
Nov 03 00:05:06.529980 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.529880    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9pbs\" (UniqueName: \"kubernetes.io/projected/689b30e8-936e-49a4-a775-4ce908ad1ab8-kube-api-access-h9pbs\") pod \"service-ca-operator-544d749c8c-w6k5p\" (UID: \"689b30e8-936e-49a4-a775-4ce908ad1ab8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p"
Nov 03 00:05:06.529980 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.529897    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kgwx\" (UniqueName: \"kubernetes.io/projected/d8606abd-d458-4bee-b150-461becfbb83b-kube-api-access-4kgwx\") pod \"kas-connection-checker-685895c7c9-lnk6s\" (UID: \"d8606abd-d458-4bee-b150-461becfbb83b\") " pod="kube-system/kas-connection-checker-685895c7c9-lnk6s"
Nov 03 00:05:06.530232 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.529992    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgqk7\" (UniqueName: \"kubernetes.io/projected/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-kube-api-access-rgqk7\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:06.530232 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.530048    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9rtk\" (UniqueName: \"kubernetes.io/projected/667223ca-ffbc-4182-beb6-67e68116171f-kube-api-access-t9rtk\") pod \"network-check-source-6f78d59f99-8mjkf\" (UID: \"667223ca-ffbc-4182-beb6-67e68116171f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8mjkf"
Nov 03 00:05:06.530232 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.530064    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr45n\" (UniqueName: \"kubernetes.io/projected/eddf03ef-53ea-4fc0-9f3f-da7d741cf06a-kube-api-access-wr45n\") pod \"volume-data-source-validator-58c9f5f46d-v4sr6\" (UID: \"eddf03ef-53ea-4fc0-9f3f-da7d741cf06a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v4sr6"
Nov 03 00:05:06.530232 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.530111    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw7ww\" (UniqueName: \"kubernetes.io/projected/c1de1047-c4c9-4d0b-8f72-bafc79987c2e-kube-api-access-kw7ww\") pod \"console-operator-7cd56b8dbc-5d95d\" (UID: \"c1de1047-c4c9-4d0b-8f72-bafc79987c2e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:06.531246 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.531225    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnl7p\" (UniqueName: \"kubernetes.io/projected/a8dde726-eabf-405e-9d38-a110274353ca-kube-api-access-jnl7p\") pod \"cluster-samples-operator-6ddb6c8b94-hcrh6\" (UID: \"a8dde726-eabf-405e-9d38-a110274353ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"
Nov 03 00:05:06.555635 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.555599    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc"
Nov 03 00:05:06.558591 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.558570    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29894400-s5xp5"
Nov 03 00:05:06.586582 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.586547    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-lnk6s"
Nov 03 00:05:06.605121 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.603441    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:06.613190 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.610254    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-tmp-dir\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:06.613190 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.610809    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:06.613190 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.610857    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-config-volume\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:06.613190 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.610885    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6w5f\" (UniqueName: \"kubernetes.io/projected/735e8903-5dfe-4cb2-8937-cfc3d09ffa36-kube-api-access-r6w5f\") pod \"ingress-canary-n68xd\" (UID: \"735e8903-5dfe-4cb2-8937-cfc3d09ffa36\") " pod="openshift-ingress-canary/ingress-canary-n68xd"
Nov 03 00:05:06.613190 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:06.611046    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 00:05:06.613190 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:06.611065    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-668bbfbdd4-9m8pn: secret "image-registry-tls" not found
Nov 03 00:05:06.613190 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:06.611129    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls podName:4d7b91ff-203c-4e03-9ad7-9b60215b61b6 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:07.111109661 +0000 UTC m=+31.868882233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls") pod "image-registry-668bbfbdd4-9m8pn" (UID: "4d7b91ff-203c-4e03-9ad7-9b60215b61b6") : secret "image-registry-tls" not found
Nov 03 00:05:06.613190 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.610368    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:06.613190 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.611761    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-tmp-dir\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:06.613190 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.612506    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-config-volume\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:06.613190 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.612554    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2m6bp\" (UID: \"bb5193c7-10f4-4436-9c69-20e6087439c2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2m6bp"
Nov 03 00:05:06.613190 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.612601    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-installation-pull-secrets\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:06.613190 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.612654    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-trusted-ca\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:06.613190 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.612683    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbzll\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-kube-api-access-bbzll\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:06.613190 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.612719    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-ca-trust-extracted\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:06.613190 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.612734    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bb5193c7-10f4-4436-9c69-20e6087439c2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2m6bp\" (UID: \"bb5193c7-10f4-4436-9c69-20e6087439c2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2m6bp"
Nov 03 00:05:06.613190 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.612771    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-image-registry-private-configuration\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:06.614309 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.612789    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-bound-sa-token\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:06.614309 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.612827    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-certificates\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:06.614309 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.612841    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkvnm\" (UniqueName: \"kubernetes.io/projected/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-kube-api-access-dkvnm\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:06.614309 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.612856    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/735e8903-5dfe-4cb2-8937-cfc3d09ffa36-cert\") pod \"ingress-canary-n68xd\" (UID: \"735e8903-5dfe-4cb2-8937-cfc3d09ffa36\") " pod="openshift-ingress-canary/ingress-canary-n68xd"
Nov 03 00:05:06.614309 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.612872    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:06.614309 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:06.612988    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 00:05:06.614309 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:06.613046    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls podName:7cdb4c55-92a6-4791-8ab8-fcb22ef3a145 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:07.113030753 +0000 UTC m=+31.870803321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls") pod "dns-default-lcdlg" (UID: "7cdb4c55-92a6-4791-8ab8-fcb22ef3a145") : secret "dns-default-metrics-tls" not found
Nov 03 00:05:06.614309 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:06.613104    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 00:05:06.614309 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:06.613138    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert podName:bb5193c7-10f4-4436-9c69-20e6087439c2 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:07.113127465 +0000 UTC m=+31.870900032 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2m6bp" (UID: "bb5193c7-10f4-4436-9c69-20e6087439c2") : secret "networking-console-plugin-cert" not found
Nov 03 00:05:06.614896 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.614397    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-certificates\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:06.616591 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.615545    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-ca-trust-extracted\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:06.616591 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.615992    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-installation-pull-secrets\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:06.616591 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.616137    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-trusted-ca\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:06.616591 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.616511    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bb5193c7-10f4-4436-9c69-20e6087439c2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2m6bp\" (UID: \"bb5193c7-10f4-4436-9c69-20e6087439c2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2m6bp"
Nov 03 00:05:06.624943 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.623868    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-image-registry-private-configuration\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:06.626159 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.626103    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/735e8903-5dfe-4cb2-8937-cfc3d09ffa36-cert\") pod \"ingress-canary-n68xd\" (UID: \"735e8903-5dfe-4cb2-8937-cfc3d09ffa36\") " pod="openshift-ingress-canary/ingress-canary-n68xd"
Nov 03 00:05:06.644340 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.644300    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkvnm\" (UniqueName: \"kubernetes.io/projected/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-kube-api-access-dkvnm\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:06.647891 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.647855    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbzll\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-kube-api-access-bbzll\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:06.650169 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.650136    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-bound-sa-token\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:06.652213 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.652136    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6w5f\" (UniqueName: \"kubernetes.io/projected/735e8903-5dfe-4cb2-8937-cfc3d09ffa36-kube-api-access-r6w5f\") pod \"ingress-canary-n68xd\" (UID: \"735e8903-5dfe-4cb2-8937-cfc3d09ffa36\") " pod="openshift-ingress-canary/ingress-canary-n68xd"
Nov 03 00:05:06.667126 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.667089    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v4sr6"
Nov 03 00:05:06.677887 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.677849    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8mjkf"
Nov 03 00:05:06.709084 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.708785    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p"
Nov 03 00:05:06.735519 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.735492    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n68xd"
Nov 03 00:05:06.798979 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.795230    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:05:06.798979 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.795721    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:05:06.798979 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.796076    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:05:06.810999 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.810413    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dlnxq\""
Nov 03 00:05:06.810999 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.810604    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 00:05:06.810999 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.810699    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8v79z\""
Nov 03 00:05:06.810999 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.810750    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 00:05:06.810999 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.810839    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wsqcw\""
Nov 03 00:05:06.841310 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.841037    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2mp9l"]
Nov 03 00:05:06.855104 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.852409    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lt4f5"]
Nov 03 00:05:06.859757 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:05:06.859723    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod913d1e5a_3dfc_4282_b5f0_95c4484b31da.slice/crio-33492d80a3afb785c34967b66d6b9022171cd7cc706d7549e07017fca5bca917 WatchSource:0}: Error finding container 33492d80a3afb785c34967b66d6b9022171cd7cc706d7549e07017fca5bca917: Status 404 returned error can't find the container with id 33492d80a3afb785c34967b66d6b9022171cd7cc706d7549e07017fca5bca917
Nov 03 00:05:06.905783 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.905756    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc"]
Nov 03 00:05:06.908389 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:05:06.908355    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b66660_5608_4e82_a1eb_db3d6ce57ca5.slice/crio-fe36277fd7b6c5722e4848adb1f4e278fff67e03f4a714cb3d478fb71d58794c WatchSource:0}: Error finding container fe36277fd7b6c5722e4848adb1f4e278fff67e03f4a714cb3d478fb71d58794c: Status 404 returned error can't find the container with id fe36277fd7b6c5722e4848adb1f4e278fff67e03f4a714cb3d478fb71d58794c
Nov 03 00:05:06.918791 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.918682    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29894400-s5xp5"]
Nov 03 00:05:06.921632 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.921606    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:06.921632 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:05:06.921618    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa3da9a_3a5f_49a4_8a09_0036fdc5aa44.slice/crio-e534f8d6a73d1ff175ee0a5a2695bd877777cb30a483c171cafda7c5946ace57 WatchSource:0}: Error finding container e534f8d6a73d1ff175ee0a5a2695bd877777cb30a483c171cafda7c5946ace57: Status 404 returned error can't find the container with id e534f8d6a73d1ff175ee0a5a2695bd877777cb30a483c171cafda7c5946ace57
Nov 03 00:05:06.921813 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.921741    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:06.921813 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.921764    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffkwv\" (UID: \"dbea6432-5ba3-49f3-8a83-db2605a3aa2c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"
Nov 03 00:05:06.921813 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:06.921783    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle podName:12eb002e-9765-4443-b488-6cec02b9c961 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:07.921761005 +0000 UTC m=+32.679533575 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle") pod "router-default-7df847f57-khf2f" (UID: "12eb002e-9765-4443-b488-6cec02b9c961") : configmap references non-existent config key: service-ca.crt
Nov 03 00:05:06.921984 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:06.921840    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 00:05:06.921984 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.921855    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:06.921984 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:06.921870    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 00:05:06.921984 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:06.921875    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls podName:dbea6432-5ba3-49f3-8a83-db2605a3aa2c nodeName:}" failed. No retries permitted until 2026-11-03 00:05:07.921865304 +0000 UTC m=+32.679637858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ffkwv" (UID: "dbea6432-5ba3-49f3-8a83-db2605a3aa2c") : secret "cluster-monitoring-operator-tls" not found
Nov 03 00:05:06.921984 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:06.921903    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs podName:12eb002e-9765-4443-b488-6cec02b9c961 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:07.921893634 +0000 UTC m=+32.679666210 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs") pod "router-default-7df847f57-khf2f" (UID: "12eb002e-9765-4443-b488-6cec02b9c961") : secret "router-metrics-certs-default" not found
Nov 03 00:05:06.922182 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:06.922007    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 00:05:06.922182 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:06.922021    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7564dc7d49-qtdkm: secret "image-registry-tls" not found
Nov 03 00:05:06.922182 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:06.922072    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls podName:ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:07.922056311 +0000 UTC m=+32.679828865 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls") pod "image-registry-7564dc7d49-qtdkm" (UID: "ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3") : secret "image-registry-tls" not found
Nov 03 00:05:06.965566 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.965490    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lnk6s"]
Nov 03 00:05:06.972145 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:06.972120    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-kzsm6"]
Nov 03 00:05:06.978424 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:05:06.978387    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d4ed33_0500_4c65_ba72_b2a584b6bdf2.slice/crio-064e4777f893298302176dc608810e2c15662c38066e2bacbb86c206a1fdfa35 WatchSource:0}: Error finding container 064e4777f893298302176dc608810e2c15662c38066e2bacbb86c206a1fdfa35: Status 404 returned error can't find the container with id 064e4777f893298302176dc608810e2c15662c38066e2bacbb86c206a1fdfa35
Nov 03 00:05:07.005969 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:05:07.005933    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1de1047_c4c9_4d0b_8f72_bafc79987c2e.slice/crio-3e48db05aa0f14e58da60aba11b7801ff25fac95d8b6797d39edd78abdf2b4ee WatchSource:0}: Error finding container 3e48db05aa0f14e58da60aba11b7801ff25fac95d8b6797d39edd78abdf2b4ee: Status 404 returned error can't find the container with id 3e48db05aa0f14e58da60aba11b7801ff25fac95d8b6797d39edd78abdf2b4ee
Nov 03 00:05:07.021399 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:07.021362    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5d95d"]
Nov 03 00:05:07.023129 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:07.023102    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hcrh6\" (UID: \"a8dde726-eabf-405e-9d38-a110274353ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"
Nov 03 00:05:07.023316 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:07.023296    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 00:05:07.023380 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:07.023358    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls podName:a8dde726-eabf-405e-9d38-a110274353ca nodeName:}" failed. No retries permitted until 2026-11-03 00:05:08.023339062 +0000 UTC m=+32.781111618 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hcrh6" (UID: "a8dde726-eabf-405e-9d38-a110274353ca") : secret "samples-operator-tls" not found
Nov 03 00:05:07.036352 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:07.036283    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v4sr6"]
Nov 03 00:05:07.037883 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:07.036627    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-lnk6s" event={"ID":"d8606abd-d458-4bee-b150-461becfbb83b","Type":"ContainerStarted","Data":"d0e33ec8758b4860c3c56bb4804adbb0957f25001607aa4efca11dcb808febb3"}
Nov 03 00:05:07.039042 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:07.038955    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-kzsm6" event={"ID":"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2","Type":"ContainerStarted","Data":"064e4777f893298302176dc608810e2c15662c38066e2bacbb86c206a1fdfa35"}
Nov 03 00:05:07.042603 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:05:07.042562    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeddf03ef_53ea_4fc0_9f3f_da7d741cf06a.slice/crio-d310053e8d88c0ce69bc8a9f7d169268a0d1b263840568fe264b8105f1815dce WatchSource:0}: Error finding container d310053e8d88c0ce69bc8a9f7d169268a0d1b263840568fe264b8105f1815dce: Status 404 returned error can't find the container with id d310053e8d88c0ce69bc8a9f7d169268a0d1b263840568fe264b8105f1815dce
Nov 03 00:05:07.047166 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:07.047083    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29894400-s5xp5" event={"ID":"daa3da9a-3a5f-49a4-8a09-0036fdc5aa44","Type":"ContainerStarted","Data":"e534f8d6a73d1ff175ee0a5a2695bd877777cb30a483c171cafda7c5946ace57"}
Nov 03 00:05:07.056551 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:07.056446    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc" event={"ID":"38b66660-5608-4e82-a1eb-db3d6ce57ca5","Type":"ContainerStarted","Data":"fe36277fd7b6c5722e4848adb1f4e278fff67e03f4a714cb3d478fb71d58794c"}
Nov 03 00:05:07.070147 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:07.069264    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-lt4f5" event={"ID":"913d1e5a-3dfc-4282-b5f0-95c4484b31da","Type":"ContainerStarted","Data":"33492d80a3afb785c34967b66d6b9022171cd7cc706d7549e07017fca5bca917"}
Nov 03 00:05:07.070147 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:05:07.069452    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667223ca_ffbc_4182_beb6_67e68116171f.slice/crio-7dbc28b60bb8f7ec99da45a1dfa4c162f3c620de1a64f8d9e6c094405e4ee516 WatchSource:0}: Error finding container 7dbc28b60bb8f7ec99da45a1dfa4c162f3c620de1a64f8d9e6c094405e4ee516: Status 404 returned error can't find the container with id 7dbc28b60bb8f7ec99da45a1dfa4c162f3c620de1a64f8d9e6c094405e4ee516
Nov 03 00:05:07.070795 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:07.070658    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-8mjkf"]
Nov 03 00:05:07.074768 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:07.074735    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d" event={"ID":"c1de1047-c4c9-4d0b-8f72-bafc79987c2e","Type":"ContainerStarted","Data":"3e48db05aa0f14e58da60aba11b7801ff25fac95d8b6797d39edd78abdf2b4ee"}
Nov 03 00:05:07.076252 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:07.076227    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-2mp9l" event={"ID":"ee7974f8-92b4-498a-bf54-2c3a353f4b49","Type":"ContainerStarted","Data":"e3354abe343c02c13547e14d64d5169f536b3fa0bc46cfd92bd9b306ed7d40a5"}
Nov 03 00:05:07.095945 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:07.095103    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p"]
Nov 03 00:05:07.104051 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:07.104020    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n68xd"]
Nov 03 00:05:07.109774 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:05:07.109730    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod689b30e8_936e_49a4_a775_4ce908ad1ab8.slice/crio-3f3628bbb595515569d3baf7515603db2ec1f972bfaf52e530cda709833793d2 WatchSource:0}: Error finding container 3f3628bbb595515569d3baf7515603db2ec1f972bfaf52e530cda709833793d2: Status 404 returned error can't find the container with id 3f3628bbb595515569d3baf7515603db2ec1f972bfaf52e530cda709833793d2
Nov 03 00:05:07.113700 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:05:07.113663    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735e8903_5dfe_4cb2_8937_cfc3d09ffa36.slice/crio-e601c2fff52128b9a39f52622375e33e465c0b247a12cfec41cc65c556022d1e WatchSource:0}: Error finding container e601c2fff52128b9a39f52622375e33e465c0b247a12cfec41cc65c556022d1e: Status 404 returned error can't find the container with id e601c2fff52128b9a39f52622375e33e465c0b247a12cfec41cc65c556022d1e
Nov 03 00:05:07.124888 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:07.124855    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:07.125070 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:07.124920    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:07.125070 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:07.124956    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2m6bp\" (UID: \"bb5193c7-10f4-4436-9c69-20e6087439c2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2m6bp"
Nov 03 00:05:07.125182 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:07.125122    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 00:05:07.125182 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:07.125172    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert podName:bb5193c7-10f4-4436-9c69-20e6087439c2 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:08.12515954 +0000 UTC m=+32.882932107 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2m6bp" (UID: "bb5193c7-10f4-4436-9c69-20e6087439c2") : secret "networking-console-plugin-cert" not found
Nov 03 00:05:07.125283 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:07.125218    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 00:05:07.125283 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:07.125238    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls podName:7cdb4c55-92a6-4791-8ab8-fcb22ef3a145 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:08.125231639 +0000 UTC m=+32.883004204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls") pod "dns-default-lcdlg" (UID: "7cdb4c55-92a6-4791-8ab8-fcb22ef3a145") : secret "dns-default-metrics-tls" not found
Nov 03 00:05:07.125380 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:07.125293    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 00:05:07.125380 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:07.125302    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-668bbfbdd4-9m8pn: secret "image-registry-tls" not found
Nov 03 00:05:07.125380 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:07.125321    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls podName:4d7b91ff-203c-4e03-9ad7-9b60215b61b6 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:08.125315734 +0000 UTC m=+32.883088299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls") pod "image-registry-668bbfbdd4-9m8pn" (UID: "4d7b91ff-203c-4e03-9ad7-9b60215b61b6") : secret "image-registry-tls" not found
Nov 03 00:05:07.934075 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:07.934022    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:07.934286 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:07.934219    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:07.934286 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:07.934251    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffkwv\" (UID: \"dbea6432-5ba3-49f3-8a83-db2605a3aa2c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"
Nov 03 00:05:07.934413 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:07.934295    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:07.935235 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:07.934465    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 00:05:07.935235 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:07.934484    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7564dc7d49-qtdkm: secret "image-registry-tls" not found
Nov 03 00:05:07.935235 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:07.934545    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls podName:ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:09.934525426 +0000 UTC m=+34.692297983 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls") pod "image-registry-7564dc7d49-qtdkm" (UID: "ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3") : secret "image-registry-tls" not found
Nov 03 00:05:07.935235 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:07.935014    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle podName:12eb002e-9765-4443-b488-6cec02b9c961 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:09.934998266 +0000 UTC m=+34.692770832 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle") pod "router-default-7df847f57-khf2f" (UID: "12eb002e-9765-4443-b488-6cec02b9c961") : configmap references non-existent config key: service-ca.crt
Nov 03 00:05:07.935235 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:07.935090    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 00:05:07.935235 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:07.935122    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs podName:12eb002e-9765-4443-b488-6cec02b9c961 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:09.935112228 +0000 UTC m=+34.692884782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs") pod "router-default-7df847f57-khf2f" (UID: "12eb002e-9765-4443-b488-6cec02b9c961") : secret "router-metrics-certs-default" not found
Nov 03 00:05:07.935235 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:07.935171    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 00:05:07.935235 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:07.935200    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls podName:dbea6432-5ba3-49f3-8a83-db2605a3aa2c nodeName:}" failed. No retries permitted until 2026-11-03 00:05:09.935190889 +0000 UTC m=+34.692963443 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ffkwv" (UID: "dbea6432-5ba3-49f3-8a83-db2605a3aa2c") : secret "cluster-monitoring-operator-tls" not found
Nov 03 00:05:08.035238 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:08.035193    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hcrh6\" (UID: \"a8dde726-eabf-405e-9d38-a110274353ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"
Nov 03 00:05:08.035990 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:08.035481    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 00:05:08.035990 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:08.035557    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls podName:a8dde726-eabf-405e-9d38-a110274353ca nodeName:}" failed. No retries permitted until 2026-11-03 00:05:10.035534555 +0000 UTC m=+34.793307113 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hcrh6" (UID: "a8dde726-eabf-405e-9d38-a110274353ca") : secret "samples-operator-tls" not found
Nov 03 00:05:08.090008 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:08.089955    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-2mp9l" event={"ID":"ee7974f8-92b4-498a-bf54-2c3a353f4b49","Type":"ContainerStarted","Data":"b28adb173e52feee61be47560815a9fbc3b6b800606831d390776fa2364f16ca"}
Nov 03 00:05:08.097959 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:08.097825    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n68xd" event={"ID":"735e8903-5dfe-4cb2-8937-cfc3d09ffa36","Type":"ContainerStarted","Data":"e601c2fff52128b9a39f52622375e33e465c0b247a12cfec41cc65c556022d1e"}
Nov 03 00:05:08.101069 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:08.101029    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p" event={"ID":"689b30e8-936e-49a4-a775-4ce908ad1ab8","Type":"ContainerStarted","Data":"3f3628bbb595515569d3baf7515603db2ec1f972bfaf52e530cda709833793d2"}
Nov 03 00:05:08.111955 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:08.111234    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-lnk6s" event={"ID":"d8606abd-d458-4bee-b150-461becfbb83b","Type":"ContainerStarted","Data":"2d932ef09389c1ec7ba142c7450236b2837675f28c81a06832f523ebbc864205"}
Nov 03 00:05:08.117647 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:08.117536    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8mjkf" event={"ID":"667223ca-ffbc-4182-beb6-67e68116171f","Type":"ContainerStarted","Data":"7dbc28b60bb8f7ec99da45a1dfa4c162f3c620de1a64f8d9e6c094405e4ee516"}
Nov 03 00:05:08.126799 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:08.126662    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v4sr6" event={"ID":"eddf03ef-53ea-4fc0-9f3f-da7d741cf06a","Type":"ContainerStarted","Data":"d310053e8d88c0ce69bc8a9f7d169268a0d1b263840568fe264b8105f1815dce"}
Nov 03 00:05:08.134181 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:08.133835    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-2mp9l" podStartSLOduration=434.133815634 podStartE2EDuration="7m14.133815634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:08.132011795 +0000 UTC m=+32.889784371" watchObservedRunningTime="2026-11-03 00:05:08.133815634 +0000 UTC m=+32.891588210"
Nov 03 00:05:08.137932 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:08.136900    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:08.137932 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:08.136985    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:08.137932 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:08.137041    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2m6bp\" (UID: \"bb5193c7-10f4-4436-9c69-20e6087439c2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2m6bp"
Nov 03 00:05:08.137932 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:08.137221    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 00:05:08.137932 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:08.137288    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert podName:bb5193c7-10f4-4436-9c69-20e6087439c2 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:10.137268616 +0000 UTC m=+34.895041171 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2m6bp" (UID: "bb5193c7-10f4-4436-9c69-20e6087439c2") : secret "networking-console-plugin-cert" not found
Nov 03 00:05:08.137932 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:08.137701    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 00:05:08.137932 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:08.137750    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls podName:7cdb4c55-92a6-4791-8ab8-fcb22ef3a145 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:10.137733929 +0000 UTC m=+34.895506482 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls") pod "dns-default-lcdlg" (UID: "7cdb4c55-92a6-4791-8ab8-fcb22ef3a145") : secret "dns-default-metrics-tls" not found
Nov 03 00:05:08.137932 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:08.137817    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 00:05:08.137932 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:08.137828    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-668bbfbdd4-9m8pn: secret "image-registry-tls" not found
Nov 03 00:05:08.137932 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:08.137861    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls podName:4d7b91ff-203c-4e03-9ad7-9b60215b61b6 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:10.137850159 +0000 UTC m=+34.895622715 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls") pod "image-registry-668bbfbdd4-9m8pn" (UID: "4d7b91ff-203c-4e03-9ad7-9b60215b61b6") : secret "image-registry-tls" not found
Nov 03 00:05:08.139798 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:08.139117    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29894400-s5xp5" event={"ID":"daa3da9a-3a5f-49a4-8a09-0036fdc5aa44","Type":"ContainerStarted","Data":"c31fe2c21e01834528f35f724afb86b8577a5b769c8caf26daa715d4c4c00c0a"}
Nov 03 00:05:08.144764 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:08.144177    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-lt4f5" event={"ID":"913d1e5a-3dfc-4282-b5f0-95c4484b31da","Type":"ContainerStarted","Data":"10c9085c74aa5e902dcca0548d918ec5ed47b3b48cc9197b45ddaae41c1f0792"}
Nov 03 00:05:08.171431 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:08.170816    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-lnk6s" podStartSLOduration=434.170795923 podStartE2EDuration="7m14.170795923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:08.169060143 +0000 UTC m=+32.926832718" watchObservedRunningTime="2026-11-03 00:05:08.170795923 +0000 UTC m=+32.928568496"
Nov 03 00:05:08.247779 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:08.246761    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-pruner-29894400-s5xp5" podStartSLOduration=308.246741205 podStartE2EDuration="5m8.246741205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:08.209546511 +0000 UTC m=+32.967319085" watchObservedRunningTime="2026-11-03 00:05:08.246741205 +0000 UTC m=+33.004513781"
Nov 03 00:05:09.657750 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:09.657709    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs\") pod \"network-metrics-daemon-phwfk\" (UID: \"a173095c-7c94-4533-8529-81b464886f05\") " pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:05:09.658287 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:09.657831    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7dvp\" (UniqueName: \"kubernetes.io/projected/d683af01-29f3-4634-ae6d-3de8a5c4fd85-kube-api-access-k7dvp\") pod \"network-check-target-ttc95\" (UID: \"d683af01-29f3-4634-ae6d-3de8a5c4fd85\") " pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:05:09.658287 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:09.657883    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6aecdeeb-4efb-413e-a26c-9270468062a8-original-pull-secret\") pod \"global-pull-secret-syncer-bgpx8\" (UID: \"6aecdeeb-4efb-413e-a26c-9270468062a8\") " pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:05:09.658287 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:09.657919    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 00:05:09.658287 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:09.658003    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs podName:a173095c-7c94-4533-8529-81b464886f05 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:41.657980534 +0000 UTC m=+66.415753091 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs") pod "network-metrics-daemon-phwfk" (UID: "a173095c-7c94-4533-8529-81b464886f05") : secret "metrics-daemon-secret" not found
Nov 03 00:05:09.666373 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:09.666336    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6aecdeeb-4efb-413e-a26c-9270468062a8-original-pull-secret\") pod \"global-pull-secret-syncer-bgpx8\" (UID: \"6aecdeeb-4efb-413e-a26c-9270468062a8\") " pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:05:09.667875 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:09.667845    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7dvp\" (UniqueName: \"kubernetes.io/projected/d683af01-29f3-4634-ae6d-3de8a5c4fd85-kube-api-access-k7dvp\") pod \"network-check-target-ttc95\" (UID: \"d683af01-29f3-4634-ae6d-3de8a5c4fd85\") " pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:05:09.917188 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:09.917080    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:05:09.940284 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:09.940245    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:05:09.961453 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:09.961410    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:09.961627 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:09.961562    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:09.961627 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:09.961592    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffkwv\" (UID: \"dbea6432-5ba3-49f3-8a83-db2605a3aa2c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"
Nov 03 00:05:09.961722 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:09.961629    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:09.961800 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:09.961782    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 00:05:09.961846 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:09.961806    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7564dc7d49-qtdkm: secret "image-registry-tls" not found
Nov 03 00:05:09.961894 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:09.961869    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls podName:ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:13.961848406 +0000 UTC m=+38.719620976 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls") pod "image-registry-7564dc7d49-qtdkm" (UID: "ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3") : secret "image-registry-tls" not found
Nov 03 00:05:09.962362 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:09.962340    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle podName:12eb002e-9765-4443-b488-6cec02b9c961 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:13.962319349 +0000 UTC m=+38.720091907 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle") pod "router-default-7df847f57-khf2f" (UID: "12eb002e-9765-4443-b488-6cec02b9c961") : configmap references non-existent config key: service-ca.crt
Nov 03 00:05:09.962498 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:09.962434    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 00:05:09.962498 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:09.962476    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs podName:12eb002e-9765-4443-b488-6cec02b9c961 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:13.962464145 +0000 UTC m=+38.720236697 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs") pod "router-default-7df847f57-khf2f" (UID: "12eb002e-9765-4443-b488-6cec02b9c961") : secret "router-metrics-certs-default" not found
Nov 03 00:05:09.962618 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:09.962528    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 00:05:09.962618 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:09.962563    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls podName:dbea6432-5ba3-49f3-8a83-db2605a3aa2c nodeName:}" failed. No retries permitted until 2026-11-03 00:05:13.962552092 +0000 UTC m=+38.720324646 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ffkwv" (UID: "dbea6432-5ba3-49f3-8a83-db2605a3aa2c") : secret "cluster-monitoring-operator-tls" not found
Nov 03 00:05:10.063008 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:10.062540    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hcrh6\" (UID: \"a8dde726-eabf-405e-9d38-a110274353ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"
Nov 03 00:05:10.063008 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:10.062800    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 00:05:10.063008 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:10.062870    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls podName:a8dde726-eabf-405e-9d38-a110274353ca nodeName:}" failed. No retries permitted until 2026-11-03 00:05:14.062849249 +0000 UTC m=+38.820621807 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hcrh6" (UID: "a8dde726-eabf-405e-9d38-a110274353ca") : secret "samples-operator-tls" not found
Nov 03 00:05:10.155814 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:10.155771    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v4sr6" event={"ID":"eddf03ef-53ea-4fc0-9f3f-da7d741cf06a","Type":"ContainerStarted","Data":"5b5fb8496af169e3f03b9efee83db423365eb67299ee39f804521b23ef1d4d7f"}
Nov 03 00:05:10.163177 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:10.163138    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2m6bp\" (UID: \"bb5193c7-10f4-4436-9c69-20e6087439c2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2m6bp"
Nov 03 00:05:10.163441 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:10.163330    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 00:05:10.163441 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:10.163420    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert podName:bb5193c7-10f4-4436-9c69-20e6087439c2 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:14.163399386 +0000 UTC m=+38.921171943 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2m6bp" (UID: "bb5193c7-10f4-4436-9c69-20e6087439c2") : secret "networking-console-plugin-cert" not found
Nov 03 00:05:10.163569 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:10.163467    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:10.163569 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:10.163530    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:10.163662 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:10.163627    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 00:05:10.163708 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:10.163663    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 00:05:10.163708 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:10.163678    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-668bbfbdd4-9m8pn: secret "image-registry-tls" not found
Nov 03 00:05:10.163779 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:10.163728    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls podName:7cdb4c55-92a6-4791-8ab8-fcb22ef3a145 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:14.163706531 +0000 UTC m=+38.921479105 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls") pod "dns-default-lcdlg" (UID: "7cdb4c55-92a6-4791-8ab8-fcb22ef3a145") : secret "dns-default-metrics-tls" not found
Nov 03 00:05:10.163779 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:10.163745    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls podName:4d7b91ff-203c-4e03-9ad7-9b60215b61b6 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:14.163738833 +0000 UTC m=+38.921511386 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls") pod "image-registry-668bbfbdd4-9m8pn" (UID: "4d7b91ff-203c-4e03-9ad7-9b60215b61b6") : secret "image-registry-tls" not found
Nov 03 00:05:10.194614 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:10.192950    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-lt4f5" podStartSLOduration=436.192931661 podStartE2EDuration="7m16.192931661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:08.24767093 +0000 UTC m=+33.005443505" watchObservedRunningTime="2026-11-03 00:05:10.192931661 +0000 UTC m=+34.950704236"
Nov 03 00:05:10.266127 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:10.265951    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v4sr6" podStartSLOduration=404.249773781 podStartE2EDuration="6m46.265902329s" totalImagesPullingTime="2.016128548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:10.193511683 +0000 UTC m=+34.951284258" watchObservedRunningTime="2026-11-03 00:05:10.265902329 +0000 UTC m=+35.023674904"
Nov 03 00:05:10.278682 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:10.278649    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ttc95"]
Nov 03 00:05:10.308996 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:10.308965    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bgpx8"]
Nov 03 00:05:10.316544 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:05:10.316505    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aecdeeb_4efb_413e_a26c_9270468062a8.slice/crio-e98a161f0c761579d58a105c20911bdaae023367f88ffc6be5f4dafd85a3fa77 WatchSource:0}: Error finding container e98a161f0c761579d58a105c20911bdaae023367f88ffc6be5f4dafd85a3fa77: Status 404 returned error can't find the container with id e98a161f0c761579d58a105c20911bdaae023367f88ffc6be5f4dafd85a3fa77
Nov 03 00:05:11.161600 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:11.161552    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ttc95" event={"ID":"d683af01-29f3-4634-ae6d-3de8a5c4fd85","Type":"ContainerStarted","Data":"dd42c6e0fad9c6df6a09e53e66c81812205707ec597690d24a3739ea61411c03"}
Nov 03 00:05:11.163958 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:11.163889    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bgpx8" event={"ID":"6aecdeeb-4efb-413e-a26c-9270468062a8","Type":"ContainerStarted","Data":"e98a161f0c761579d58a105c20911bdaae023367f88ffc6be5f4dafd85a3fa77"}
Nov 03 00:05:12.170637 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:12.170601    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/0.log"
Nov 03 00:05:12.171119 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:12.170651    2574 generic.go:356] "Generic (PLEG): container finished" podID="c1de1047-c4c9-4d0b-8f72-bafc79987c2e" containerID="d734ee21762ce89e98b797e2851db848a9e34d9628410955140e819a347cace3" exitCode=255
Nov 03 00:05:12.171119 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:12.170787    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d" event={"ID":"c1de1047-c4c9-4d0b-8f72-bafc79987c2e","Type":"ContainerDied","Data":"d734ee21762ce89e98b797e2851db848a9e34d9628410955140e819a347cace3"}
Nov 03 00:05:12.171226 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:12.171161    2574 scope.go:122] "RemoveContainer" containerID="d734ee21762ce89e98b797e2851db848a9e34d9628410955140e819a347cace3"
Nov 03 00:05:12.174281 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:12.174219    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-kzsm6" event={"ID":"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2","Type":"ContainerStarted","Data":"4d5e7453f09dfd64250c0cd6b64caf6ebc2043881d9acba0d3b880b12da8d44a"}
Nov 03 00:05:12.176815 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:12.176781    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc" event={"ID":"38b66660-5608-4e82-a1eb-db3d6ce57ca5","Type":"ContainerStarted","Data":"e3357bdcf4cbec9868e9d50e392c3a72be4ee9619f5333683d160c5d649df639"}
Nov 03 00:05:12.256394 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:12.256281    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc" podStartSLOduration=402.436698643 podStartE2EDuration="6m47.256260759s" totalImagesPullingTime="4.819562116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:12.254842661 +0000 UTC m=+37.012615239" watchObservedRunningTime="2026-11-03 00:05:12.256260759 +0000 UTC m=+37.014033334"
Nov 03 00:05:12.301787 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:12.301228    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-kzsm6" podStartSLOduration=404.593298368 podStartE2EDuration="6m49.301204354s" totalImagesPullingTime="4.707905986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:12.300427583 +0000 UTC m=+37.058200159" watchObservedRunningTime="2026-11-03 00:05:12.301204354 +0000 UTC m=+37.058976930"
Nov 03 00:05:12.865359 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:12.865307    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xwdjk"]
Nov 03 00:05:12.890501 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:12.890460    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-94gh8"]
Nov 03 00:05:12.893339 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:12.893295    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:12.907635 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:12.907162    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xwdjk"]
Nov 03 00:05:12.909307 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:12.908331    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-94gh8"
Nov 03 00:05:12.931065 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:12.931025    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-94gh8"]
Nov 03 00:05:13.002050 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.002016    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 00:05:13.002261 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.002118    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 00:05:13.002261 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.002153    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 00:05:13.002261 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.002204    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bnks\" (UniqueName: \"kubernetes.io/projected/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-api-access-6bnks\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:13.002433 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.002270    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4v7ss\""
Nov 03 00:05:13.002433 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.002031    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 00:05:13.002433 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.002275    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7llq5\""
Nov 03 00:05:13.002433 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.002282    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86d13ddf-e3f4-49ad-a779-b9590f50076d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:13.002613 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.002446    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7b5n\" (UniqueName: \"kubernetes.io/projected/4c5d55cb-0306-4dc1-b25b-c2444f148af6-kube-api-access-x7b5n\") pod \"migrator-66df5d496b-94gh8\" (UID: \"4c5d55cb-0306-4dc1-b25b-c2444f148af6\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-94gh8"
Nov 03 00:05:13.002613 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.002560    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/86d13ddf-e3f4-49ad-a779-b9590f50076d-crio-socket\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:13.002704 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.002622    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:13.002704 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.002678    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/86d13ddf-e3f4-49ad-a779-b9590f50076d-data-volume\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:13.103725 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.103689    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/86d13ddf-e3f4-49ad-a779-b9590f50076d-crio-socket\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:13.103926 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.103737    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:13.103926 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.103773    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/86d13ddf-e3f4-49ad-a779-b9590f50076d-data-volume\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:13.103926 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.103804    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bnks\" (UniqueName: \"kubernetes.io/projected/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-api-access-6bnks\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:13.103926 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.103850    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86d13ddf-e3f4-49ad-a779-b9590f50076d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:13.103926 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.103889    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7b5n\" (UniqueName: \"kubernetes.io/projected/4c5d55cb-0306-4dc1-b25b-c2444f148af6-kube-api-access-x7b5n\") pod \"migrator-66df5d496b-94gh8\" (UID: \"4c5d55cb-0306-4dc1-b25b-c2444f148af6\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-94gh8"
Nov 03 00:05:13.104271 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.104251    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/86d13ddf-e3f4-49ad-a779-b9590f50076d-crio-socket\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:13.104339 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:13.104320    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 00:05:13.104394 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:13.104374    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-rbac-proxy-cm podName:86d13ddf-e3f4-49ad-a779-b9590f50076d nodeName:}" failed. No retries permitted until 2026-11-03 00:05:13.604354519 +0000 UTC m=+38.362127087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xwdjk" (UID: "86d13ddf-e3f4-49ad-a779-b9590f50076d") : configmap "kube-rbac-proxy" not found
Nov 03 00:05:13.104867 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.104848    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/86d13ddf-e3f4-49ad-a779-b9590f50076d-data-volume\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:13.105182 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:13.105169    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:13.105240 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:13.105209    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86d13ddf-e3f4-49ad-a779-b9590f50076d-insights-runtime-extractor-tls podName:86d13ddf-e3f4-49ad-a779-b9590f50076d nodeName:}" failed. No retries permitted until 2026-11-03 00:05:13.605197976 +0000 UTC m=+38.362970542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86d13ddf-e3f4-49ad-a779-b9590f50076d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xwdjk" (UID: "86d13ddf-e3f4-49ad-a779-b9590f50076d") : secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:13.126157 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.126119    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bnks\" (UniqueName: \"kubernetes.io/projected/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-api-access-6bnks\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:13.126649 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.126622    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7b5n\" (UniqueName: \"kubernetes.io/projected/4c5d55cb-0306-4dc1-b25b-c2444f148af6-kube-api-access-x7b5n\") pod \"migrator-66df5d496b-94gh8\" (UID: \"4c5d55cb-0306-4dc1-b25b-c2444f148af6\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-94gh8"
Nov 03 00:05:13.184641 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.184607    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/1.log"
Nov 03 00:05:13.185265 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.185176    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/0.log"
Nov 03 00:05:13.185265 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.185224    2574 generic.go:356] "Generic (PLEG): container finished" podID="c1de1047-c4c9-4d0b-8f72-bafc79987c2e" containerID="82c6612b250a969062f81eafeb8c6730b0ebdea3d3d3cc15571a64b5c5a49b0b" exitCode=255
Nov 03 00:05:13.185385 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.185278    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d" event={"ID":"c1de1047-c4c9-4d0b-8f72-bafc79987c2e","Type":"ContainerDied","Data":"82c6612b250a969062f81eafeb8c6730b0ebdea3d3d3cc15571a64b5c5a49b0b"}
Nov 03 00:05:13.185385 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.185352    2574 scope.go:122] "RemoveContainer" containerID="d734ee21762ce89e98b797e2851db848a9e34d9628410955140e819a347cace3"
Nov 03 00:05:13.185635 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.185614    2574 scope.go:122] "RemoveContainer" containerID="82c6612b250a969062f81eafeb8c6730b0ebdea3d3d3cc15571a64b5c5a49b0b"
Nov 03 00:05:13.185871 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:13.185850    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5d95d_openshift-console-operator(c1de1047-c4c9-4d0b-8f72-bafc79987c2e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d" podUID="c1de1047-c4c9-4d0b-8f72-bafc79987c2e"
Nov 03 00:05:13.187813 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.187763    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n68xd" event={"ID":"735e8903-5dfe-4cb2-8937-cfc3d09ffa36","Type":"ContainerStarted","Data":"e27b1f66053c3531ea342134cd2145cfc25d571a2d35c106eaf9b21b70cbc106"}
Nov 03 00:05:13.190627 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.190591    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p" event={"ID":"689b30e8-936e-49a4-a775-4ce908ad1ab8","Type":"ContainerStarted","Data":"e83ccef58b1c16a55c030ca8700573bcba68bd1ef3943055a38bff4bbfe75a38"}
Nov 03 00:05:13.263858 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.263785    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-n68xd" podStartSLOduration=1.864896902 podStartE2EDuration="7.263760977s" totalImagesPullingTime="5.398864075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:13.262037721 +0000 UTC m=+38.019810294" watchObservedRunningTime="2026-11-03 00:05:13.263760977 +0000 UTC m=+38.021533552"
Nov 03 00:05:13.282935 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.282661    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-94gh8"
Nov 03 00:05:13.300933 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.300279    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p" podStartSLOduration=404.886204631 podStartE2EDuration="6m50.300253593s" totalImagesPullingTime="5.414048962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:13.298751029 +0000 UTC m=+38.056523620" watchObservedRunningTime="2026-11-03 00:05:13.300253593 +0000 UTC m=+38.058026168"
Nov 03 00:05:13.610372 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.610336    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:13.610590 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:13.610485    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 00:05:13.610590 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.610495    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86d13ddf-e3f4-49ad-a779-b9590f50076d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:13.610590 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:13.610571    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-rbac-proxy-cm podName:86d13ddf-e3f4-49ad-a779-b9590f50076d nodeName:}" failed. No retries permitted until 2026-11-03 00:05:14.61055052 +0000 UTC m=+39.368323079 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xwdjk" (UID: "86d13ddf-e3f4-49ad-a779-b9590f50076d") : configmap "kube-rbac-proxy" not found
Nov 03 00:05:13.610745 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:13.610620    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:13.610745 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:13.610660    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86d13ddf-e3f4-49ad-a779-b9590f50076d-insights-runtime-extractor-tls podName:86d13ddf-e3f4-49ad-a779-b9590f50076d nodeName:}" failed. No retries permitted until 2026-11-03 00:05:14.610647768 +0000 UTC m=+39.368420340 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86d13ddf-e3f4-49ad-a779-b9590f50076d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xwdjk" (UID: "86d13ddf-e3f4-49ad-a779-b9590f50076d") : secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:13.689207 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:13.688978    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-94gh8"]
Nov 03 00:05:13.696292 ip-10-0-133-161 kubenswrapper[2574]: W1103 00:05:13.696249    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c5d55cb_0306_4dc1_b25b_c2444f148af6.slice/crio-fdaa2ff297d604e60bdd287263f647e7383635a2e8c3eccef5fbd2e80e83b271 WatchSource:0}: Error finding container fdaa2ff297d604e60bdd287263f647e7383635a2e8c3eccef5fbd2e80e83b271: Status 404 returned error can't find the container with id fdaa2ff297d604e60bdd287263f647e7383635a2e8c3eccef5fbd2e80e83b271
Nov 03 00:05:14.015130 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:14.015034    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:14.015130 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:14.015101    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffkwv\" (UID: \"dbea6432-5ba3-49f3-8a83-db2605a3aa2c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"
Nov 03 00:05:14.015440 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:14.015155    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:14.015440 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:14.015206    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:14.015440 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:14.015219    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 00:05:14.015440 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:14.015233    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 00:05:14.015440 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:14.015323    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs podName:12eb002e-9765-4443-b488-6cec02b9c961 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:22.015299552 +0000 UTC m=+46.773072118 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs") pod "router-default-7df847f57-khf2f" (UID: "12eb002e-9765-4443-b488-6cec02b9c961") : secret "router-metrics-certs-default" not found
Nov 03 00:05:14.015440 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:14.015343    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 00:05:14.015440 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:14.015363    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7564dc7d49-qtdkm: secret "image-registry-tls" not found
Nov 03 00:05:14.015440 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:14.015350    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle podName:12eb002e-9765-4443-b488-6cec02b9c961 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:22.01533424 +0000 UTC m=+46.773107031 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle") pod "router-default-7df847f57-khf2f" (UID: "12eb002e-9765-4443-b488-6cec02b9c961") : configmap references non-existent config key: service-ca.crt
Nov 03 00:05:14.015440 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:14.015422    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls podName:dbea6432-5ba3-49f3-8a83-db2605a3aa2c nodeName:}" failed. No retries permitted until 2026-11-03 00:05:22.015411084 +0000 UTC m=+46.773183639 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ffkwv" (UID: "dbea6432-5ba3-49f3-8a83-db2605a3aa2c") : secret "cluster-monitoring-operator-tls" not found
Nov 03 00:05:14.015440 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:14.015439    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls podName:ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:22.015428661 +0000 UTC m=+46.773201214 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls") pod "image-registry-7564dc7d49-qtdkm" (UID: "ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3") : secret "image-registry-tls" not found
Nov 03 00:05:14.116187 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:14.116147    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hcrh6\" (UID: \"a8dde726-eabf-405e-9d38-a110274353ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"
Nov 03 00:05:14.116367 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:14.116344    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 00:05:14.116441 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:14.116432    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls podName:a8dde726-eabf-405e-9d38-a110274353ca nodeName:}" failed. No retries permitted until 2026-11-03 00:05:22.116415814 +0000 UTC m=+46.874188366 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hcrh6" (UID: "a8dde726-eabf-405e-9d38-a110274353ca") : secret "samples-operator-tls" not found
Nov 03 00:05:14.196299 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:14.196256    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ttc95" event={"ID":"d683af01-29f3-4634-ae6d-3de8a5c4fd85","Type":"ContainerStarted","Data":"2a9d18e8c973ef11edbf0048bdef0b32931a251ac7f3c574bf567e5ab3422c5b"}
Nov 03 00:05:14.197066 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:14.197042    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:05:14.199448 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:14.199386    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8mjkf" event={"ID":"667223ca-ffbc-4182-beb6-67e68116171f","Type":"ContainerStarted","Data":"3a93f2fcaae62325cc5160f4787a1fc7245dcce4a333d9cc637188b7c844a371"}
Nov 03 00:05:14.201383 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:14.201330    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-94gh8" event={"ID":"4c5d55cb-0306-4dc1-b25b-c2444f148af6","Type":"ContainerStarted","Data":"fdaa2ff297d604e60bdd287263f647e7383635a2e8c3eccef5fbd2e80e83b271"}
Nov 03 00:05:14.203258 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:14.203236    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/1.log"
Nov 03 00:05:14.203779 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:14.203760    2574 scope.go:122] "RemoveContainer" containerID="82c6612b250a969062f81eafeb8c6730b0ebdea3d3d3cc15571a64b5c5a49b0b"
Nov 03 00:05:14.204044 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:14.204022    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5d95d_openshift-console-operator(c1de1047-c4c9-4d0b-8f72-bafc79987c2e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d" podUID="c1de1047-c4c9-4d0b-8f72-bafc79987c2e"
Nov 03 00:05:14.217357 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:14.217162    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:14.217357 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:14.217261    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 00:05:14.217357 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:14.217323    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls podName:7cdb4c55-92a6-4791-8ab8-fcb22ef3a145 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:22.217303687 +0000 UTC m=+46.975076253 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls") pod "dns-default-lcdlg" (UID: "7cdb4c55-92a6-4791-8ab8-fcb22ef3a145") : secret "dns-default-metrics-tls" not found
Nov 03 00:05:14.217659 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:14.217444    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:14.217659 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:14.217523    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2m6bp\" (UID: \"bb5193c7-10f4-4436-9c69-20e6087439c2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2m6bp"
Nov 03 00:05:14.217659 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:14.217658    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 00:05:14.217800 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:14.217670    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-668bbfbdd4-9m8pn: secret "image-registry-tls" not found
Nov 03 00:05:14.217800 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:14.217702    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls podName:4d7b91ff-203c-4e03-9ad7-9b60215b61b6 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:22.217691909 +0000 UTC m=+46.975464460 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls") pod "image-registry-668bbfbdd4-9m8pn" (UID: "4d7b91ff-203c-4e03-9ad7-9b60215b61b6") : secret "image-registry-tls" not found
Nov 03 00:05:14.217800 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:14.217747    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 00:05:14.217800 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:14.217795    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert podName:bb5193c7-10f4-4436-9c69-20e6087439c2 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:22.217780369 +0000 UTC m=+46.975552926 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2m6bp" (UID: "bb5193c7-10f4-4436-9c69-20e6087439c2") : secret "networking-console-plugin-cert" not found
Nov 03 00:05:14.233903 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:14.233832    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ttc95" podStartSLOduration=35.881767606 podStartE2EDuration="39.233817672s" totalImagesPullingTime="3.352050066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:14.232419237 +0000 UTC m=+38.990191812" watchObservedRunningTime="2026-11-03 00:05:14.233817672 +0000 UTC m=+38.991590246"
Nov 03 00:05:14.269989 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:14.269854    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8mjkf" podStartSLOduration=401.954217578 podStartE2EDuration="6m48.269833852s" totalImagesPullingTime="6.315616274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:58:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:14.269703639 +0000 UTC m=+39.027476216" watchObservedRunningTime="2026-11-03 00:05:14.269833852 +0000 UTC m=+39.027606429"
Nov 03 00:05:14.622084 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:14.622043    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86d13ddf-e3f4-49ad-a779-b9590f50076d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:14.622272 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:14.622167    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:14.622272 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:14.622230    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:14.622375 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:14.622286    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 00:05:14.622375 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:14.622318    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86d13ddf-e3f4-49ad-a779-b9590f50076d-insights-runtime-extractor-tls podName:86d13ddf-e3f4-49ad-a779-b9590f50076d nodeName:}" failed. No retries permitted until 2026-11-03 00:05:16.622297049 +0000 UTC m=+41.380069605 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86d13ddf-e3f4-49ad-a779-b9590f50076d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xwdjk" (UID: "86d13ddf-e3f4-49ad-a779-b9590f50076d") : secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:14.622375 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:14.622340    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-rbac-proxy-cm podName:86d13ddf-e3f4-49ad-a779-b9590f50076d nodeName:}" failed. No retries permitted until 2026-11-03 00:05:16.622330267 +0000 UTC m=+41.380102823 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xwdjk" (UID: "86d13ddf-e3f4-49ad-a779-b9590f50076d") : configmap "kube-rbac-proxy" not found
Nov 03 00:05:16.214023 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:16.213976    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-94gh8" event={"ID":"4c5d55cb-0306-4dc1-b25b-c2444f148af6","Type":"ContainerStarted","Data":"2c68e3576f6934e87512659e1811b970f767cdc340644edfb3580849ed044b3a"}
Nov 03 00:05:16.250647 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:16.250551    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-94gh8" podStartSLOduration=2.839206036 podStartE2EDuration="4.250534485s" totalImagesPullingTime="1.411328449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:16.250282067 +0000 UTC m=+41.008054642" watchObservedRunningTime="2026-11-03 00:05:16.250534485 +0000 UTC m=+41.008307060"
Nov 03 00:05:16.610628 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:16.610587    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:16.610807 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:16.610684    2574 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:16.611098 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:16.611083    2574 scope.go:122] "RemoveContainer" containerID="82c6612b250a969062f81eafeb8c6730b0ebdea3d3d3cc15571a64b5c5a49b0b"
Nov 03 00:05:16.611296 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:16.611275    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5d95d_openshift-console-operator(c1de1047-c4c9-4d0b-8f72-bafc79987c2e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d" podUID="c1de1047-c4c9-4d0b-8f72-bafc79987c2e"
Nov 03 00:05:16.642617 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:16.642570    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:16.642799 ip-10-0-133-161 kubenswrapper[2574]: I1103 00:05:16.642745    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86d13ddf-e3f4-49ad-a779-b9590f50076d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:16.642799 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:16.642761    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 00:05:16.642886 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:16.642848    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-rbac-proxy-cm podName:86d13ddf-e3f4-49ad-a779-b9590f50076d nodeName:}" failed. No retries permitted until 2026-11-03 00:05:20.642826251 +0000 UTC m=+45.400598819 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xwdjk" (UID: "86d13ddf-e3f4-49ad-a779-b9590f50076d") : configmap "kube-rbac-proxy" not found
Nov 03 00:05:16.642886 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:16.642865    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:16.643013 ip-10-0-133-161 kubenswrapper[2574]: E1103 00:05:16.642929    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86d13ddf-e3f4-49ad-a779-b9590f50076d-insights-runtime-extractor-tls podName:86d13ddf-e3f4-49ad-a779-b9590f50076d nodeName:}" failed. No retries permitted until 2026-11-03 00:05:20.642892695 +0000 UTC m=+45.400665268 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86d13ddf-e3f4-49ad-a779-b9590f50076d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xwdjk" (UID: "86d13ddf-e3f4-49ad-a779-b9590f50076d") : secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:17.025632 ip-10-0-133-161 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 00:05:17.042264 ip-10-0-133-161 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 00:05:17.042432 ip-10-0-133-161 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 00:05:17.042477 ip-10-0-133-161 systemd[1]: kubelet.service: Consumed 3.182s CPU time.
Nov 03 00:05:17.057604 ip-10-0-133-161 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 00:05:17.149754 ip-10-0-133-161 kubenswrapper[5012]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 00:05:17.149754 ip-10-0-133-161 kubenswrapper[5012]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 00:05:17.149754 ip-10-0-133-161 kubenswrapper[5012]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 00:05:17.149754 ip-10-0-133-161 kubenswrapper[5012]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 00:05:17.152104 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152016    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:05:17.152104 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152100    5012 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:05:17.152104 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152106    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:05:17.152104 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152110    5012 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:05:17.152375 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152114    5012 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:05:17.152375 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152119    5012 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:05:17.152375 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152123    5012 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:05:17.152375 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152127    5012 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:05:17.152375 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152131    5012 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:05:17.152375 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152135    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:05:17.152375 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152139    5012 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:05:17.152375 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152143    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:05:17.152375 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152147    5012 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:05:17.152375 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152158    5012 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:05:17.152375 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152163    5012 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:05:17.152375 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152167    5012 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:05:17.152375 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152171    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:05:17.152375 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152175    5012 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:05:17.152375 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152179    5012 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:05:17.152375 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152183    5012 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:05:17.152375 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152187    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:05:17.152375 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152191    5012 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:05:17.152375 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152195    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:05:17.152375 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152199    5012 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:05:17.153264 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152203    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:05:17.153264 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152206    5012 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:05:17.153264 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152210    5012 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:05:17.153264 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152214    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:05:17.153264 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152218    5012 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:05:17.153264 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152222    5012 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:05:17.153264 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152226    5012 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:05:17.153264 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152231    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:05:17.153264 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152235    5012 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:05:17.153264 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152241    5012 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:05:17.153264 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152245    5012 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:05:17.153264 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152253    5012 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:05:17.153264 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152260    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:05:17.153264 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152265    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:05:17.153264 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152270    5012 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:05:17.153264 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152274    5012 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:05:17.153264 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152278    5012 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:05:17.153264 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152282    5012 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:05:17.154134 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152287    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:05:17.154134 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152291    5012 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:05:17.154134 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152296    5012 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:05:17.154134 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152300    5012 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:05:17.154134 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152305    5012 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:05:17.154134 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152310    5012 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:05:17.154134 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152314    5012 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:05:17.154134 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152319    5012 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:05:17.154134 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152323    5012 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:05:17.154134 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152327    5012 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:05:17.154134 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152332    5012 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:05:17.154134 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152336    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:05:17.154134 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152340    5012 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:05:17.154134 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152345    5012 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:05:17.154134 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152350    5012 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:05:17.154134 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152354    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:05:17.154134 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152358    5012 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:05:17.154134 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152362    5012 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:05:17.154134 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152367    5012 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:05:17.155003 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152372    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:05:17.155003 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152376    5012 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:05:17.155003 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152381    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:05:17.155003 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152386    5012 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:05:17.155003 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152391    5012 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:05:17.155003 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152395    5012 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:05:17.155003 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152399    5012 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:05:17.155003 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152404    5012 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:05:17.155003 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152408    5012 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:05:17.155003 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152412    5012 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:05:17.155003 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152416    5012 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:05:17.155003 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152421    5012 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:05:17.155003 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152427    5012 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:05:17.155003 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152433    5012 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:05:17.155003 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152439    5012 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:05:17.155003 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152445    5012 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:05:17.155003 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152449    5012 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:05:17.155003 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152454    5012 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:05:17.155003 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152459    5012 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:05:17.155003 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152463    5012 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:05:17.155920 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152468    5012 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:05:17.155920 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152472    5012 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:05:17.155920 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152477    5012 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:05:17.155920 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152481    5012 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:05:17.155920 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152485    5012 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:05:17.155920 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152490    5012 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:05:17.155920 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152494    5012 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:05:17.155920 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152498    5012 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:05:17.155920 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152503    5012 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:05:17.155920 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152507    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:05:17.155920 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152512    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:05:17.155920 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152516    5012 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:05:17.155920 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152521    5012 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:05:17.155920 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152525    5012 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:05:17.155920 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152530    5012 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:05:17.155920 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152535    5012 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:05:17.155920 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152539    5012 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:05:17.155920 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152543    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:05:17.155920 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152548    5012 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:05:17.155920 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152552    5012 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:05:17.156838 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152556    5012 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:05:17.156838 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152560    5012 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:05:17.156838 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152565    5012 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:05:17.156838 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152569    5012 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:05:17.156838 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.152576    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:05:17.156838 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.153877    5012 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:05:17.156838 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.153890    5012 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:05:17.156838 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.153895    5012 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:05:17.156838 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.153900    5012 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:05:17.156838 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.153904    5012 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:05:17.156838 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.153927    5012 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:05:17.156838 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.153933    5012 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:05:17.156838 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.153938    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:05:17.156838 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.153942    5012 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:05:17.156838 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.153946    5012 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:05:17.156838 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.153950    5012 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:05:17.156838 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.153955    5012 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:05:17.156838 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.153959    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:05:17.156838 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.153962    5012 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:05:17.156838 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.153968    5012 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:05:17.157743 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.153973    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:05:17.157743 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.153977    5012 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:05:17.157743 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.153981    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:05:17.157743 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.153986    5012 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:05:17.157743 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.153990    5012 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:05:17.157743 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.153994    5012 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:05:17.157743 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.153998    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:05:17.157743 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154002    5012 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:05:17.157743 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154007    5012 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:05:17.157743 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154011    5012 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:05:17.157743 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154016    5012 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:05:17.157743 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154020    5012 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:05:17.157743 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154024    5012 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:05:17.157743 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154029    5012 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:05:17.157743 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154033    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:05:17.157743 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154037    5012 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:05:17.157743 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154042    5012 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:05:17.157743 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154046    5012 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:05:17.157743 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154051    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:05:17.157743 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154055    5012 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:05:17.158633 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154060    5012 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:05:17.158633 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154064    5012 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:05:17.158633 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154068    5012 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:05:17.158633 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154073    5012 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:05:17.158633 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154078    5012 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:05:17.158633 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154082    5012 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:05:17.158633 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154087    5012 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:05:17.158633 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154091    5012 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:05:17.158633 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154096    5012 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:05:17.158633 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154100    5012 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:05:17.158633 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154105    5012 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:05:17.158633 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154109    5012 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:05:17.158633 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154113    5012 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:05:17.158633 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154117    5012 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:05:17.158633 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154122    5012 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:05:17.158633 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154129    5012 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:05:17.158633 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154135    5012 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:05:17.158633 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154140    5012 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:05:17.158633 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154144    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:05:17.159197 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154149    5012 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:05:17.159197 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154153    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:05:17.159197 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154157    5012 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:05:17.159197 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154162    5012 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:05:17.159197 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154167    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:05:17.159197 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154171    5012 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:05:17.159197 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154176    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:05:17.159197 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154180    5012 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:05:17.159197 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154185    5012 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:05:17.159197 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154189    5012 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:05:17.159197 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154194    5012 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:05:17.159197 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154198    5012 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:05:17.159197 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154202    5012 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:05:17.159197 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154206    5012 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:05:17.159197 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154216    5012 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:05:17.159197 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154221    5012 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:05:17.159197 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154226    5012 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:05:17.159197 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154231    5012 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:05:17.159197 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154238    5012 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:05:17.159661 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154244    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:05:17.159661 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154250    5012 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:05:17.159661 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154255    5012 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:05:17.159661 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154260    5012 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:05:17.159661 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154264    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:05:17.159661 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154269    5012 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:05:17.159661 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154273    5012 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:05:17.159661 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154277    5012 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:05:17.159661 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154281    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:05:17.159661 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154286    5012 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:05:17.159661 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154290    5012 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:05:17.159661 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154294    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:05:17.159661 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154298    5012 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:05:17.159661 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154303    5012 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:05:17.159661 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154307    5012 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:05:17.159661 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154312    5012 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:05:17.159661 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154316    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:05:17.159661 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154320    5012 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:05:17.159661 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154325    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:05:17.160161 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154329    5012 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:05:17.160161 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154334    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:05:17.160161 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154338    5012 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:05:17.160161 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154343    5012 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:05:17.160161 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154347    5012 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:05:17.160161 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154352    5012 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:05:17.160161 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154356    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:05:17.160161 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154361    5012 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:05:17.160161 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154367    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:05:17.160161 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154372    5012 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:05:17.160161 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154376    5012 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:05:17.160161 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154380    5012 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:05:17.160161 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154386    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:05:17.160161 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.154399    5012 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:05:17.160161 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155231    5012 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:05:17.160161 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155241    5012 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:05:17.160161 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155245    5012 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:05:17.160161 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155250    5012 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:05:17.160161 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155254    5012 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:05:17.160161 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155262    5012 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:05:17.160726 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155268    5012 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:05:17.160726 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155275    5012 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:05:17.160726 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155279    5012 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:05:17.160726 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155284    5012 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:05:17.160726 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155293    5012 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:05:17.160726 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155298    5012 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:05:17.160726 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155303    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:05:17.160726 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155308    5012 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:05:17.160726 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155312    5012 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:05:17.160726 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155316    5012 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:05:17.160726 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155320    5012 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:05:17.160726 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155325    5012 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:05:17.160726 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155329    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:05:17.160726 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155333    5012 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:05:17.160726 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155337    5012 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:05:17.160726 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155342    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:05:17.160726 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155346    5012 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:05:17.160726 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155350    5012 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:05:17.160726 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155355    5012 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:05:17.161220 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155359    5012 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:05:17.161220 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155366    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:05:17.161220 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155371    5012 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:05:17.161220 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155375    5012 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:05:17.161220 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155380    5012 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:05:17.161220 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155384    5012 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:05:17.161220 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155399    5012 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:05:17.161220 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155404    5012 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:05:17.161220 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155408    5012 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:05:17.161220 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155412    5012 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:05:17.161220 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155417    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:05:17.161220 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155421    5012 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:05:17.161220 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155425    5012 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:05:17.161220 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155429    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:05:17.161220 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155434    5012 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:05:17.161220 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155438    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:05:17.161220 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155442    5012 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:05:17.161220 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155446    5012 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:05:17.161220 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155450    5012 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:05:17.161220 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155455    5012 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:05:17.161765 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155459    5012 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:05:17.161765 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155463    5012 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:05:17.161765 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155467    5012 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:05:17.161765 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155472    5012 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:05:17.161765 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155476    5012 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:05:17.161765 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155480    5012 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:05:17.161765 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155485    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:05:17.161765 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155489    5012 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:05:17.161765 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155493    5012 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:05:17.161765 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155498    5012 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:05:17.161765 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155504    5012 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:05:17.161765 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155508    5012 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:05:17.161765 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155512    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:05:17.161765 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155518    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:05:17.161765 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155523    5012 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:05:17.161765 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155527    5012 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:05:17.161765 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155532    5012 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:05:17.161765 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155536    5012 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:05:17.161765 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155540    5012 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:05:17.162258 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155553    5012 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:05:17.162258 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155560    5012 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:05:17.162258 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155565    5012 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:05:17.162258 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155569    5012 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:05:17.162258 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155574    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:05:17.162258 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155578    5012 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:05:17.162258 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155583    5012 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:05:17.162258 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155587    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:05:17.162258 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155592    5012 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:05:17.162258 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155596    5012 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:05:17.162258 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155601    5012 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:05:17.162258 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155605    5012 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:05:17.162258 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155609    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:05:17.162258 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155613    5012 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:05:17.162258 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155617    5012 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:05:17.162258 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155621    5012 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:05:17.162258 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155626    5012 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:05:17.162258 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155631    5012 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:05:17.162258 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155635    5012 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:05:17.162258 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155639    5012 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:05:17.162758 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155643    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:05:17.162758 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155648    5012 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:05:17.162758 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155652    5012 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:05:17.162758 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155656    5012 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:05:17.162758 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155660    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:05:17.162758 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155664    5012 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:05:17.162758 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155671    5012 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:05:17.162758 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155676    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:05:17.162758 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155680    5012 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:05:17.162758 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155684    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:05:17.162758 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155688    5012 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:05:17.162758 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155692    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:05:17.162758 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155696    5012 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:05:17.162758 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155708    5012 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:05:17.162758 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155713    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:05:17.162758 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155717    5012 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:05:17.162758 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155721    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:05:17.162758 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155726    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:05:17.162758 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155730    5012 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:05:17.162758 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155735    5012 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:05:17.163573 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155740    5012 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:05:17.163573 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.155744    5012 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:05:17.163573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.155894    5012 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 00:05:17.163573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.155925    5012 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 00:05:17.163573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.155934    5012 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 00:05:17.163573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.155941    5012 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 00:05:17.163573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.155948    5012 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 00:05:17.163573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.155953    5012 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 00:05:17.163573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.155960    5012 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 00:05:17.163573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.155966    5012 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 00:05:17.163573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.155972    5012 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 00:05:17.163573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.155977    5012 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 00:05:17.163573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.155982    5012 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 00:05:17.163573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.155988    5012 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 00:05:17.163573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.155993    5012 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 00:05:17.163573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.155997    5012 flags.go:64] FLAG: --cgroup-root=""
Nov 03 00:05:17.163573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156002    5012 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 00:05:17.163573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156007    5012 flags.go:64] FLAG: --client-ca-file=""
Nov 03 00:05:17.163573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156012    5012 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 00:05:17.163573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156018    5012 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 00:05:17.163573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156030    5012 flags.go:64] FLAG: --cluster-domain=""
Nov 03 00:05:17.163573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156034    5012 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 00:05:17.163573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156039    5012 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 00:05:17.163573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156044    5012 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 00:05:17.164353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156050    5012 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 00:05:17.164353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156056    5012 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 00:05:17.164353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156062    5012 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 00:05:17.164353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156067    5012 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 00:05:17.164353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156073    5012 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 00:05:17.164353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156077    5012 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 00:05:17.164353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156082    5012 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 00:05:17.164353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156087    5012 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 00:05:17.164353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156092    5012 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 00:05:17.164353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156096    5012 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 00:05:17.164353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156104    5012 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 00:05:17.164353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156109    5012 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 00:05:17.164353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156114    5012 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 00:05:17.164353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156119    5012 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 00:05:17.164353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156123    5012 flags.go:64] FLAG: --enable-server="true"
Nov 03 00:05:17.164353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156128    5012 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 00:05:17.164353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156138    5012 flags.go:64] FLAG: --event-burst="100"
Nov 03 00:05:17.164353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156144    5012 flags.go:64] FLAG: --event-qps="50"
Nov 03 00:05:17.164353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156148    5012 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 00:05:17.164353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156153    5012 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 00:05:17.164353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156158    5012 flags.go:64] FLAG: --eviction-hard=""
Nov 03 00:05:17.164353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156167    5012 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 00:05:17.164353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156172    5012 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 00:05:17.164353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156177    5012 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 00:05:17.164353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156182    5012 flags.go:64] FLAG: --eviction-soft=""
Nov 03 00:05:17.165010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156188    5012 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 00:05:17.165010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156193    5012 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 00:05:17.165010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156199    5012 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 00:05:17.165010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156204    5012 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 00:05:17.165010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156208    5012 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 00:05:17.165010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156213    5012 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 00:05:17.165010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156218    5012 flags.go:64] FLAG: --feature-gates=""
Nov 03 00:05:17.165010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156224    5012 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 00:05:17.165010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156229    5012 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 00:05:17.165010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156235    5012 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 00:05:17.165010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156240    5012 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 00:05:17.165010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156246    5012 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 00:05:17.165010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156251    5012 flags.go:64] FLAG: --help="false"
Nov 03 00:05:17.165010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156256    5012 flags.go:64] FLAG: --hostname-override="ip-10-0-133-161.ec2.internal"
Nov 03 00:05:17.165010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156262    5012 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 00:05:17.165010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156266    5012 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 00:05:17.165010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156272    5012 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 00:05:17.165010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156277    5012 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 00:05:17.165010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156283    5012 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 00:05:17.165010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156288    5012 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 00:05:17.165010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156293    5012 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 00:05:17.165010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156297    5012 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 00:05:17.165010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156302    5012 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 00:05:17.165569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156307    5012 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 00:05:17.165569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156312    5012 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 00:05:17.165569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156317    5012 flags.go:64] FLAG: --kube-reserved=""
Nov 03 00:05:17.165569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156322    5012 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 00:05:17.165569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156327    5012 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 00:05:17.165569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156334    5012 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 00:05:17.165569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156339    5012 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 00:05:17.165569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156344    5012 flags.go:64] FLAG: --lock-file=""
Nov 03 00:05:17.165569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156348    5012 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 00:05:17.165569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156353    5012 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 00:05:17.165569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156359    5012 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 00:05:17.165569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156368    5012 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 00:05:17.165569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156372    5012 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 00:05:17.165569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156377    5012 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 00:05:17.165569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156382    5012 flags.go:64] FLAG: --logging-format="text"
Nov 03 00:05:17.165569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156387    5012 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 00:05:17.165569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156392    5012 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 00:05:17.165569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156397    5012 flags.go:64] FLAG: --manifest-url=""
Nov 03 00:05:17.165569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156401    5012 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 00:05:17.165569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156409    5012 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 00:05:17.165569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156414    5012 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 00:05:17.165569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156421    5012 flags.go:64] FLAG: --max-pods="110"
Nov 03 00:05:17.165569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156426    5012 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 00:05:17.165569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156430    5012 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 00:05:17.165569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156435    5012 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156440    5012 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156445    5012 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156450    5012 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156454    5012 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156465    5012 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156471    5012 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156476    5012 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156480    5012 flags.go:64] FLAG: --pod-cidr=""
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156485    5012 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156490    5012 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156495    5012 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156500    5012 flags.go:64] FLAG: --port="10250"
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156507    5012 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156512    5012 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fa2fd233044cd6c5"
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156517    5012 flags.go:64] FLAG: --qos-reserved=""
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156522    5012 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156527    5012 flags.go:64] FLAG: --register-node="true"
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156531    5012 flags.go:64] FLAG: --register-with-taints=""
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156538    5012 flags.go:64] FLAG: --registry-burst="10"
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156543    5012 flags.go:64] FLAG: --registry-qps="5"
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156547    5012 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156552    5012 flags.go:64] FLAG: --reserved-memory=""
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156558    5012 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156563    5012 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156569    5012 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 00:05:17.166192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156573    5012 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 00:05:17.166824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156578    5012 flags.go:64] FLAG: --runonce="false"
Nov 03 00:05:17.166824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156583    5012 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 00:05:17.166824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156589    5012 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 00:05:17.166824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156594    5012 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 00:05:17.166824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156598    5012 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 00:05:17.166824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156603    5012 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 00:05:17.166824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156608    5012 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 00:05:17.166824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156613    5012 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 00:05:17.166824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156618    5012 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 00:05:17.166824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156622    5012 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 00:05:17.166824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156627    5012 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 00:05:17.166824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156634    5012 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 00:05:17.166824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156639    5012 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 00:05:17.166824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156644    5012 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 00:05:17.166824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156649    5012 flags.go:64] FLAG: --system-cgroups=""
Nov 03 00:05:17.166824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156653    5012 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 00:05:17.166824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156662    5012 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 00:05:17.166824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156667    5012 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 00:05:17.166824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156674    5012 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 00:05:17.166824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156681    5012 flags.go:64] FLAG: --tls-min-version=""
Nov 03 00:05:17.166824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156686    5012 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 00:05:17.166824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156690    5012 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 00:05:17.166824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156695    5012 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 00:05:17.166824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156700    5012 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 00:05:17.166824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156705    5012 flags.go:64] FLAG: --v="2"
Nov 03 00:05:17.167442 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156711    5012 flags.go:64] FLAG: --version="false"
Nov 03 00:05:17.167442 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156718    5012 flags.go:64] FLAG: --vmodule=""
Nov 03 00:05:17.167442 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156724    5012 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 00:05:17.167442 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.156729    5012 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 00:05:17.167442 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.156970    5012 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:05:17.167442 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.156979    5012 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:05:17.167442 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.156984    5012 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:05:17.167442 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.156989    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:05:17.167442 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.156994    5012 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:05:17.167442 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.156999    5012 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:05:17.167442 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157011    5012 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:05:17.167442 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157017    5012 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:05:17.167442 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157021    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:05:17.167442 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157025    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:05:17.167442 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157030    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:05:17.167442 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157035    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:05:17.167442 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157039    5012 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:05:17.167442 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157044    5012 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:05:17.167442 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157051    5012 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:05:17.167442 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157056    5012 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:05:17.168022 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157060    5012 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:05:17.168022 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157064    5012 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:05:17.168022 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157069    5012 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:05:17.168022 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157074    5012 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:05:17.168022 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157078    5012 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:05:17.168022 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157088    5012 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:05:17.168022 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157092    5012 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:05:17.168022 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157096    5012 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:05:17.168022 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157101    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:05:17.168022 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157105    5012 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:05:17.168022 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157110    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:05:17.168022 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157136    5012 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:05:17.168022 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157142    5012 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:05:17.168022 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157147    5012 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:05:17.168022 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157152    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:05:17.168022 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157168    5012 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:05:17.168022 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157173    5012 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:05:17.168022 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157177    5012 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:05:17.168022 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157182    5012 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:05:17.168022 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157187    5012 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:05:17.168506 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157191    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:05:17.168506 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157195    5012 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:05:17.168506 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157200    5012 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:05:17.168506 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157212    5012 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:05:17.168506 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157217    5012 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:05:17.168506 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157221    5012 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:05:17.168506 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157225    5012 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:05:17.168506 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157233    5012 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:05:17.168506 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157240    5012 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:05:17.168506 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157246    5012 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:05:17.168506 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157253    5012 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:05:17.168506 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157257    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:05:17.168506 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157266    5012 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:05:17.168506 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157270    5012 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:05:17.168506 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157275    5012 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:05:17.168506 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157279    5012 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:05:17.168506 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157284    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:05:17.168506 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157290    5012 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:05:17.168506 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157294    5012 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:05:17.169076 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157299    5012 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:05:17.169076 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157303    5012 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:05:17.169076 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157307    5012 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:05:17.169076 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157312    5012 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:05:17.169076 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157317    5012 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:05:17.169076 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157322    5012 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:05:17.169076 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157326    5012 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:05:17.169076 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157331    5012 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:05:17.169076 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157335    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:05:17.169076 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157339    5012 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:05:17.169076 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157344    5012 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:05:17.169076 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157348    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:05:17.169076 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157352    5012 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:05:17.169076 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157356    5012 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:05:17.169076 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157361    5012 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:05:17.169076 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157365    5012 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:05:17.169076 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157370    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:05:17.169076 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157375    5012 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:05:17.169076 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157380    5012 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:05:17.169076 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157384    5012 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:05:17.169848 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157389    5012 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:05:17.169848 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157393    5012 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:05:17.169848 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157399    5012 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:05:17.169848 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157407    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:05:17.169848 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157412    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:05:17.169848 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157416    5012 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:05:17.169848 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157421    5012 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:05:17.169848 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157425    5012 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:05:17.169848 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157429    5012 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:05:17.169848 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157433    5012 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:05:17.169848 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157439    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:05:17.169848 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157443    5012 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:05:17.169848 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157448    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:05:17.169848 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157452    5012 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:05:17.169848 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157456    5012 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:05:17.169848 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157461    5012 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:05:17.169848 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157466    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:05:17.169848 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157470    5012 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:05:17.169848 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157487    5012 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:05:17.170340 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157493    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:05:17.170340 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157498    5012 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:05:17.170340 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157503    5012 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:05:17.170340 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157508    5012 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:05:17.170340 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157513    5012 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:05:17.170340 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157517    5012 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:05:17.170340 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157521    5012 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:05:17.170340 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157526    5012 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:05:17.170340 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157530    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:05:17.170340 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157534    5012 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:05:17.170340 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157538    5012 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:05:17.170340 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157544    5012 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:05:17.170340 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.157585    5012 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 00:05:17.170340 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157670    5012 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:05:17.170340 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157677    5012 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:05:17.170340 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157682    5012 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:05:17.170741 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157689    5012 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:05:17.170741 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157705    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:05:17.170741 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157711    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:05:17.170741 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157715    5012 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:05:17.170741 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157720    5012 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:05:17.170741 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157724    5012 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:05:17.170741 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157729    5012 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:05:17.170741 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157735    5012 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:05:17.170741 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157739    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:05:17.170741 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157744    5012 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:05:17.170741 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157748    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:05:17.170741 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157752    5012 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:05:17.170741 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157756    5012 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:05:17.170741 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157761    5012 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:05:17.170741 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157765    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:05:17.170741 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157770    5012 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:05:17.170741 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157774    5012 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:05:17.170741 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157779    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:05:17.170741 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157783    5012 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:05:17.171353 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157788    5012 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:05:17.171353 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157792    5012 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:05:17.171353 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157796    5012 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:05:17.171353 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157801    5012 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:05:17.171353 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157805    5012 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:05:17.171353 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157809    5012 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:05:17.171353 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157813    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:05:17.171353 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157818    5012 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:05:17.171353 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157823    5012 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:05:17.171353 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157828    5012 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:05:17.171353 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157833    5012 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:05:17.171353 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157837    5012 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:05:17.171353 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157842    5012 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:05:17.171353 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157846    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:05:17.171353 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157852    5012 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:05:17.171353 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157857    5012 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:05:17.171353 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157861    5012 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:05:17.171353 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157866    5012 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:05:17.171353 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157870    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:05:17.171353 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157875    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:05:17.172045 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157881    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:05:17.172045 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157886    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:05:17.172045 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157890    5012 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:05:17.172045 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157894    5012 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:05:17.172045 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157899    5012 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:05:17.172045 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157903    5012 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:05:17.172045 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157923    5012 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:05:17.172045 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157928    5012 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:05:17.172045 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157932    5012 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:05:17.172045 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157937    5012 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:05:17.172045 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157941    5012 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:05:17.172045 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157946    5012 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:05:17.172045 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157950    5012 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:05:17.172045 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157954    5012 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:05:17.172045 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157958    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:05:17.172045 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157963    5012 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:05:17.172045 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157967    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:05:17.172045 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157971    5012 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:05:17.172045 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157975    5012 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:05:17.172766 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157979    5012 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:05:17.172766 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157984    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:05:17.172766 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157989    5012 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:05:17.172766 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157994    5012 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:05:17.172766 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.157998    5012 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:05:17.172766 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158003    5012 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:05:17.172766 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158009    5012 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:05:17.172766 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158016    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:05:17.172766 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158020    5012 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:05:17.172766 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158024    5012 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:05:17.172766 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158029    5012 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:05:17.172766 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158033    5012 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:05:17.172766 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158038    5012 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:05:17.172766 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158043    5012 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:05:17.172766 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158050    5012 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:05:17.172766 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158057    5012 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:05:17.172766 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158063    5012 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:05:17.172766 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158069    5012 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:05:17.172766 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158073    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:05:17.172766 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158077    5012 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:05:17.173558 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158082    5012 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:05:17.173558 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158087    5012 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:05:17.173558 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158091    5012 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:05:17.173558 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158096    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:05:17.173558 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158100    5012 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:05:17.173558 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158105    5012 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:05:17.173558 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158109    5012 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:05:17.173558 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158114    5012 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:05:17.173558 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158118    5012 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:05:17.173558 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158122    5012 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:05:17.173558 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158126    5012 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:05:17.173558 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158130    5012 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:05:17.173558 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158135    5012 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:05:17.173558 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158139    5012 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:05:17.173558 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158143    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:05:17.173558 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158147    5012 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:05:17.173558 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158152    5012 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:05:17.173558 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158157    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:05:17.173558 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158161    5012 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:05:17.174400 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158167    5012 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:05:17.174400 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158171    5012 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:05:17.174400 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158176    5012 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:05:17.174400 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158180    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:05:17.174400 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158185    5012 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:05:17.174400 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.158191    5012 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:05:17.174400 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.162238    5012 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 00:05:17.174400 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.162254    5012 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 00:05:17.174400 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.162272    5012 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 00:05:17.174400 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.162276    5012 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 00:05:17.174400 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162353    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:05:17.174400 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162358    5012 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:05:17.174400 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162362    5012 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:05:17.174400 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162365    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:05:17.174400 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162368    5012 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:05:17.174400 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162371    5012 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:05:17.174400 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162374    5012 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:05:17.174400 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162377    5012 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:05:17.174400 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162379    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:05:17.174884 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162382    5012 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:05:17.174884 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162385    5012 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:05:17.174884 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162387    5012 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:05:17.174884 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162397    5012 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:05:17.174884 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162400    5012 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:05:17.174884 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162403    5012 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:05:17.174884 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162406    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:05:17.174884 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162409    5012 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:05:17.174884 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162411    5012 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:05:17.174884 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162414    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:05:17.174884 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162423    5012 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:05:17.174884 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162427    5012 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:05:17.174884 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162429    5012 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:05:17.174884 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162432    5012 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:05:17.174884 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162435    5012 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:05:17.174884 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162438    5012 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:05:17.174884 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162441    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:05:17.174884 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162450    5012 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:05:17.174884 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162452    5012 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:05:17.174884 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162455    5012 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:05:17.175418 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162459    5012 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:05:17.175418 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162462    5012 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:05:17.175418 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162465    5012 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:05:17.175418 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162467    5012 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:05:17.175418 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162470    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:05:17.175418 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162472    5012 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:05:17.175418 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162475    5012 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:05:17.175418 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162478    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:05:17.175418 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162480    5012 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:05:17.175418 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162483    5012 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:05:17.175418 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162485    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:05:17.175418 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162488    5012 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:05:17.175418 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162490    5012 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:05:17.175418 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162493    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:05:17.175418 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162495    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:05:17.175418 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162498    5012 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:05:17.175418 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162500    5012 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:05:17.175418 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162503    5012 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:05:17.175418 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162505    5012 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:05:17.175418 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162508    5012 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:05:17.175970 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162511    5012 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:05:17.175970 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162513    5012 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:05:17.175970 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162516    5012 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:05:17.175970 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162524    5012 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:05:17.175970 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162527    5012 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:05:17.175970 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162529    5012 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:05:17.175970 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162534    5012 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:05:17.175970 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162538    5012 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:05:17.175970 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162541    5012 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:05:17.175970 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162544    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:05:17.175970 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162547    5012 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:05:17.175970 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162549    5012 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:05:17.175970 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162553    5012 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:05:17.175970 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162555    5012 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:05:17.175970 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162558    5012 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:05:17.175970 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162560    5012 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:05:17.175970 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162563    5012 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:05:17.175970 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162566    5012 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:05:17.175970 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162570    5012 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:05:17.176439 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162573    5012 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:05:17.176439 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162576    5012 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:05:17.176439 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162579    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:05:17.176439 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162582    5012 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:05:17.176439 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162585    5012 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:05:17.176439 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162588    5012 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:05:17.176439 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162590    5012 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:05:17.176439 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162593    5012 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:05:17.176439 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162595    5012 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:05:17.176439 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162598    5012 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:05:17.176439 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162601    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:05:17.176439 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162603    5012 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:05:17.176439 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162606    5012 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:05:17.176439 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162609    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:05:17.176439 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162611    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:05:17.176439 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162615    5012 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:05:17.176439 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162618    5012 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:05:17.176439 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162626    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:05:17.176439 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162629    5012 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:05:17.176902 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162632    5012 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:05:17.176902 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162634    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:05:17.176902 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162637    5012 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:05:17.176902 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162640    5012 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:05:17.176902 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162642    5012 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:05:17.176902 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162645    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:05:17.176902 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162648    5012 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:05:17.176902 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162650    5012 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:05:17.176902 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162653    5012 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:05:17.176902 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162655    5012 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:05:17.176902 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162658    5012 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:05:17.176902 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162660    5012 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:05:17.176902 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162663    5012 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:05:17.176902 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162665    5012 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:05:17.176902 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162668    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:05:17.176902 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162671    5012 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:05:17.176902 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162673    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:05:17.176902 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162676    5012 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:05:17.176902 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.162679    5012 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:05:17.177429 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.162707    5012 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 00:05:17.177429 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.163560    5012 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:05:17.177429 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164084    5012 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:05:17.177429 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164092    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:05:17.177429 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164098    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:05:17.177429 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164103    5012 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:05:17.177429 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164108    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:05:17.177429 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164114    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:05:17.177429 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164256    5012 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:05:17.177429 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164268    5012 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:05:17.177429 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164272    5012 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:05:17.177429 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164275    5012 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:05:17.177429 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164279    5012 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:05:17.177429 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164282    5012 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:05:17.177429 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164284    5012 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:05:17.177801 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164289    5012 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:05:17.177801 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164295    5012 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:05:17.177801 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164298    5012 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:05:17.177801 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164301    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:05:17.177801 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164305    5012 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:05:17.177801 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164307    5012 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:05:17.177801 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164312    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:05:17.177801 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164315    5012 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:05:17.177801 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164317    5012 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:05:17.177801 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164320    5012 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:05:17.177801 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164323    5012 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:05:17.177801 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164326    5012 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:05:17.177801 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164328    5012 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:05:17.177801 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164331    5012 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:05:17.177801 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164334    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:05:17.177801 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164337    5012 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:05:17.177801 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164339    5012 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:05:17.177801 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164342    5012 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:05:17.177801 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164345    5012 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:05:17.178280 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164348    5012 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:05:17.178280 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164350    5012 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:05:17.178280 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164353    5012 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:05:17.178280 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164356    5012 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:05:17.178280 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164358    5012 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:05:17.178280 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164361    5012 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:05:17.178280 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164366    5012 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:05:17.178280 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164369    5012 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:05:17.178280 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164372    5012 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:05:17.178280 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164375    5012 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:05:17.178280 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164378    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:05:17.178280 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164381    5012 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:05:17.178280 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164384    5012 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:05:17.178280 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164386    5012 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:05:17.178280 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164389    5012 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:05:17.178280 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164392    5012 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:05:17.178280 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164395    5012 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:05:17.178280 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164398    5012 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:05:17.178280 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164400    5012 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:05:17.178280 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164403    5012 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:05:17.178815 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164405    5012 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:05:17.178815 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164408    5012 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:05:17.178815 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164410    5012 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:05:17.178815 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164413    5012 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:05:17.178815 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164416    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:05:17.178815 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164418    5012 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:05:17.178815 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164421    5012 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:05:17.178815 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164424    5012 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:05:17.178815 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164426    5012 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:05:17.178815 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164429    5012 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:05:17.178815 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164431    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:05:17.178815 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164434    5012 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:05:17.178815 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164439    5012 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:05:17.178815 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164442    5012 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:05:17.178815 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164444    5012 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:05:17.178815 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164447    5012 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:05:17.178815 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164450    5012 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:05:17.178815 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164452    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:05:17.178815 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164455    5012 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:05:17.178815 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164458    5012 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:05:17.179343 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164461    5012 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:05:17.179343 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164464    5012 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:05:17.179343 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164466    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:05:17.179343 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164469    5012 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:05:17.179343 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164471    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:05:17.179343 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164474    5012 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:05:17.179343 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164477    5012 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:05:17.179343 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164479    5012 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:05:17.179343 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164482    5012 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:05:17.179343 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164484    5012 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:05:17.179343 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164488    5012 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:05:17.179343 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164490    5012 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:05:17.179343 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164493    5012 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:05:17.179343 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164495    5012 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:05:17.179343 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164498    5012 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:05:17.179343 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164501    5012 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:05:17.179343 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164503    5012 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:05:17.179343 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164506    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:05:17.179343 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164509    5012 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:05:17.179343 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164511    5012 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:05:17.179343 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164514    5012 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:05:17.179862 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164517    5012 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:05:17.179862 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164519    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:05:17.179862 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164522    5012 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:05:17.179862 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164524    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:05:17.179862 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164527    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:05:17.179862 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164530    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:05:17.179862 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164532    5012 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:05:17.179862 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164535    5012 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:05:17.179862 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164537    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:05:17.179862 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164540    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:05:17.179862 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164543    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:05:17.179862 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164545    5012 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:05:17.179862 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.164578    5012 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 00:05:17.179862 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164629    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:05:17.179862 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164634    5012 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:05:17.179862 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164637    5012 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:05:17.180391 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164640    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:05:17.180391 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164643    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:05:17.180391 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164645    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:05:17.180391 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164648    5012 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:05:17.180391 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164652    5012 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:05:17.180391 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164656    5012 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:05:17.180391 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164660    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:05:17.180391 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164662    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:05:17.180391 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164665    5012 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:05:17.180391 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164667    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:05:17.180391 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164670    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:05:17.180391 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164673    5012 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:05:17.180391 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164675    5012 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:05:17.180391 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164678    5012 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:05:17.180391 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164680    5012 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:05:17.180391 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164683    5012 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:05:17.180391 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164686    5012 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:05:17.180391 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164688    5012 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:05:17.180825 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164692    5012 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:05:17.180825 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164696    5012 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:05:17.180825 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164698    5012 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:05:17.180825 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164701    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:05:17.180825 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164704    5012 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:05:17.180825 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164707    5012 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:05:17.180825 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164710    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:05:17.180825 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164712    5012 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:05:17.180825 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164715    5012 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:05:17.180825 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164717    5012 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:05:17.180825 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164720    5012 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:05:17.180825 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164723    5012 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:05:17.180825 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164725    5012 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:05:17.180825 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164728    5012 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:05:17.180825 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164730    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:05:17.180825 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164733    5012 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:05:17.180825 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164736    5012 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:05:17.180825 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164738    5012 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:05:17.180825 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164741    5012 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:05:17.181302 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164744    5012 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:05:17.181302 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164746    5012 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:05:17.181302 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164749    5012 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:05:17.181302 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164752    5012 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:05:17.181302 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164754    5012 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:05:17.181302 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164757    5012 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:05:17.181302 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164760    5012 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:05:17.181302 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164762    5012 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:05:17.181302 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164765    5012 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:05:17.181302 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164767    5012 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:05:17.181302 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164770    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:05:17.181302 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164772    5012 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:05:17.181302 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164775    5012 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:05:17.181302 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164777    5012 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:05:17.181302 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164780    5012 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:05:17.181302 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164783    5012 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:05:17.181302 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164785    5012 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:05:17.181302 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164788    5012 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:05:17.181302 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164790    5012 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:05:17.181302 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164793    5012 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:05:17.181827 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164795    5012 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:05:17.181827 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164798    5012 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:05:17.181827 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164800    5012 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:05:17.181827 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164803    5012 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:05:17.181827 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164805    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:05:17.181827 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164808    5012 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:05:17.181827 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164810    5012 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:05:17.181827 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164813    5012 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:05:17.181827 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164815    5012 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:05:17.181827 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164818    5012 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:05:17.181827 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164821    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:05:17.181827 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164824    5012 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:05:17.181827 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164828    5012 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:05:17.181827 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164830    5012 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:05:17.181827 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164833    5012 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:05:17.181827 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164836    5012 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:05:17.181827 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164838    5012 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:05:17.181827 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164841    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:05:17.181827 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164843    5012 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:05:17.181827 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164846    5012 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:05:17.182349 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164849    5012 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:05:17.182349 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164851    5012 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:05:17.182349 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164854    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:05:17.182349 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164856    5012 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:05:17.182349 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164859    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:05:17.182349 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164861    5012 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:05:17.182349 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164864    5012 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:05:17.182349 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164866    5012 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:05:17.182349 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164869    5012 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:05:17.182349 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164871    5012 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:05:17.182349 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164873    5012 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:05:17.182349 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164876    5012 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:05:17.182349 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164879    5012 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:05:17.182349 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164881    5012 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:05:17.182349 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164888    5012 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:05:17.182349 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164890    5012 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:05:17.182349 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164893    5012 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:05:17.182349 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164896    5012 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:05:17.182349 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164898    5012 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:05:17.182349 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164901    5012 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:05:17.182349 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164904    5012 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:05:17.183211 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164925    5012 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:05:17.183211 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164929    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:05:17.183211 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164932    5012 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:05:17.183211 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164934    5012 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:05:17.183211 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:17.164938    5012 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:05:17.183211 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.164996    5012 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 00:05:17.183211 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.166007    5012 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 00:05:17.183211 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.166057    5012 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 00:05:17.183211 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.166230    5012 server.go:1014] "Starting client certificate rotation"
Nov 03 00:05:17.183211 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.166371    5012 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 00:05:17.183211 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.166439    5012 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 23:59:35 +0000 UTC" deadline="2028-07-13 17:51:13.714574906 +0000 UTC"
Nov 03 00:05:17.183211 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.166486    5012 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14849h45m56.548091958s"
Nov 03 00:05:17.183211 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.166767    5012 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 00:05:17.183211 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.167932    5012 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 00:05:17.183211 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.169131    5012 log.go:25] "Validated CRI v1 runtime API"
Nov 03 00:05:17.183211 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.170877    5012 log.go:25] "Validated CRI v1 image API"
Nov 03 00:05:17.183702 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.173838    5012 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 00:05:17.183702 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.176004    5012 fs.go:135] Filesystem UUIDs: map[4406269c-e4d8-4e91-adba-ef9b125df014:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ddbd0b69-85d2-4c2f-875f-c175681b9ede:/dev/nvme0n1p3]
Nov 03 00:05:17.183785 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.176024    5012 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/064e4777f893298302176dc608810e2c15662c38066e2bacbb86c206a1fdfa35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/064e4777f893298302176dc608810e2c15662c38066e2bacbb86c206a1fdfa35/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ffb3ce37f4ea01beca43eaf7f71968d3f6bf22a280aa4e6de499ade77a4ab42/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ffb3ce37f4ea01beca43eaf7f71968d3f6bf22a280aa4e6de499ade77a4ab42/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25b8cd2ec77476e68d59d598da6f50bc0901483679ddc07512611488c8c5469e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25b8cd2ec77476e68d59d598da6f50bc0901483679ddc07512611488c8c5469e/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32925b0fda6815092f415eee6ae94fde3490062f322d962538f8ff5fcf9c953a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32925b0fda6815092f415eee6ae94fde3490062f322d962538f8ff5fcf9c953a/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33492d80a3afb785c34967b66d6b9022171cd7cc706d7549e07017fca5bca917/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33492d80a3afb785c34967b66d6b9022171cd7cc706d7549e07017fca5bca917/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e48db05aa0f14e58da60aba11b7801ff25fac95d8b6797d39edd78abdf2b4ee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e48db05aa0f14e58da60aba11b7801ff25fac95d8b6797d39edd78abdf2b4ee/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f3628bbb595515569d3baf7515603db2ec1f972bfaf52e530cda709833793d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f3628bbb595515569d3baf7515603db2ec1f972bfaf52e530cda709833793d2/userdata/shm major:0 minor:247 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7dbc28b60bb8f7ec99da45a1dfa4c162f3c620de1a64f8d9e6c094405e4ee516/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7dbc28b60bb8f7ec99da45a1dfa4c162f3c620de1a64f8d9e6c094405e4ee516/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84cd2d9a687e85e59dd66448d9c7f03e7d89bf34dab38621ced8b345f589eeb0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84cd2d9a687e85e59dd66448d9c7f03e7d89bf34dab38621ced8b345f589eeb0/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ce6a3437092f12bae141073692ecf67563bf2879195884d2fabe043160a95ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ce6a3437092f12bae141073692ecf67563bf2879195884d2fabe043160a95ec/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a22d53dec8b80ea7bce25b6229cff9d1c7b321b62c8adfb26174dc5682256fa4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a22d53dec8b80ea7bce25b6229cff9d1c7b321b62c8adfb26174dc5682256fa4/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0e33ec8758b4860c3c56bb4804adbb0957f25001607aa4efca11dcb808febb3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0e33ec8758b4860c3c56bb4804adbb0957f25001607aa4efca11dcb808febb3/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d310053e8d88c0ce69bc8a9f7d169268a0d1b263840568fe264b8105f1815dce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d310053e8d88c0ce69bc8a9f7d169268a0d1b263840568fe264b8105f1815dce/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dce57bb4de821416aa2640454e1413069f4756379b69eb399c55299f4d1d1219/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dce57bb4de821416aa2640454e1413069f4756379b69eb399c55299f4d1d1219/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd42c6e0fad9c6df6a09e53e66c81812205707ec597690d24a3739ea61411c03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd42c6e0fad9c6df6a09e53e66c81812205707ec597690d24a3739ea61411c03/userdata/shm major:0 minor:305 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ddf0b02c07944ca9b610fb313cc5817c926a312bebea337199d8e15ec25e4276/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ddf0b02c07944ca9b610fb313cc5817c926a312bebea337199d8e15ec25e4276/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3354abe343c02c13547e14d64d5169f536b3fa0bc46cfd92bd9b306ed7d40a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3354abe343c02c13547e14d64d5169f536b3fa0bc46cfd92bd9b306ed7d40a5/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e534f8d6a73d1ff175ee0a5a2695bd877777cb30a483c171cafda7c5946ace57/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e534f8d6a73d1ff175ee0a5a2695bd877777cb30a483c171cafda7c5946ace57/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e601c2fff52128b9a39f52622375e33e465c0b247a12cfec41cc65c556022d1e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e601c2fff52128b9a39f52622375e33e465c0b247a12cfec41cc65c556022d1e/userdata/shm major:0 minor:249 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e98a161f0c761579d58a105c20911bdaae023367f88ffc6be5f4dafd85a3fa77/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e98a161f0c761579d58a105c20911bdaae023367f88ffc6be5f4dafd85a3fa77/userdata/shm major:0 minor:307 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f91b5735012147ec87a0fee6fe44f7791385e421c0d95eb9e70b9d5d8ea000f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f91b5735012147ec87a0fee6fe44f7791385e421c0d95eb9e70b9d5d8ea000f2/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb6385809f85d1b7277cd035d86d0eb1b4ed29b00d661d6406ffe3bcf1d7ae20/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb6385809f85d1b7277cd035d86d0eb1b4ed29b00d661d6406ffe3bcf1d7ae20/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fdaa2ff297d604e60bdd287263f647e7383635a2e8c3eccef5fbd2e80e83b271/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fdaa2ff297d604e60bdd287263f647e7383635a2e8c3eccef5fbd2e80e83b271/userdata/shm major:0 minor:327 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe36277fd7b6c5722e4848adb1f4e278fff67e03f4a714cb3d478fb71d58794c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe36277fd7b6c5722e4848adb1f4e278fff67e03f4a714cb3d478fb71d58794c/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/015c1f9d-d4c4-4749-978a-3cd2166c7bf8/volumes/kubernetes.io~projected/kube-api-access-f59nd:{mountpoint:/var/lib/kubelet/pods/015c1f9d-d4c4-4749-978a-3cd2166c7bf8/volumes/kubernetes.io~projected/kube-api-access-f59nd major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12eb002e-9765-4443-b488-6cec02b9c961/volumes/kubernetes.io~projected/kube-api-access-5xcfc:{mountpoint:/var/lib/kubelet/pods/12eb002e-9765-4443-b488-6cec02b9c961/volumes/kubernetes.io~projected/kube-api-access-5xcfc major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12eb002e-9765-4443-b488-6cec02b9c961/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/12eb002e-9765-4443-b488-6cec02b9c961/volumes/kubernetes.io~secret/default-certificate major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12eb002e-9765-4443-b488-6cec02b9c961/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/12eb002e-9765-4443-b488-6cec02b9c961/volumes/kubernetes.io~secret/stats-auth major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28e0cef7-ba0d-4546-a193-2f411c441350/volumes/kubernetes.io~projected/kube-api-access-rmwfj:{mountpoint:/var/lib/kubelet/pods/28e0cef7-ba0d-4546-a193-2f411c441350/volumes/kubernetes.io~projected/kube-api-access-rmwfj major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38b66660-5608-4e82-a1eb-db3d6ce57ca5/volumes/kubernetes.io~projected/kube-api-access-chv2w:{mountpoint:/var/lib/kubelet/pods/38b66660-5608-4e82-a1eb-db3d6ce57ca5/volumes/kubernetes.io~projected/kube-api-access-chv2w major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38b66660-5608-4e82-a1eb-db3d6ce57ca5/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/38b66660-5608-4e82-a1eb-db3d6ce57ca5/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47258bdf-947a-4a28-86f4-02871f64941a/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/47258bdf-947a-4a28-86f4-02871f64941a/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47ea3616-f54d-4603-a56f-5c549d66eb81/volumes/kubernetes.io~projected/kube-api-access-255z4:{mountpoint:/var/lib/kubelet/pods/47ea3616-f54d-4603-a56f-5c549d66eb81/volumes/kubernetes.io~projected/kube-api-access-255z4 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c5d55cb-0306-4dc1-b25b-c2444f148af6/volumes/kubernetes.io~projected/kube-api-access-x7b5n:{mountpoint:/var/lib/kubelet/pods/4c5d55cb-0306-4dc1-b25b-c2444f148af6/volumes/kubernetes.io~projected/kube-api-access-x7b5n major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d7b91ff-203c-4e03-9ad7-9b60215b61b6/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/4d7b91ff-203c-4e03-9ad7-9b60215b61b6/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d7b91ff-203c-4e03-9ad7-9b60215b61b6/volumes/kubernetes.io~projected/kube-api-access-bbzll:{mountpoint:/var/lib/kubelet/pods/4d7b91ff-203c-4e03-9ad7-9b60215b61b6/volumes/kubernetes.io~projected/kube-api-access-bbzll major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d7b91ff-203c-4e03-9ad7-9b60215b61b6/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/4d7b91ff-203c-4e03-9ad7-9b60215b61b6/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d7b91ff-203c-4e03-9ad7-9b60215b61b6/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/4d7b91ff-203c-4e03-9ad7-9b60215b61b6/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61d5adce-3738-4cad-aace-39d9d33a739a/volumes/kubernetes.io~projected/kube-api-access-chvlq:{mountpoint:/var/lib/kubelet/pods/61d5adce-3738-4cad-aace-39d9d33a739a/volumes/kubernetes.io~projected/kube-api-access-chvlq major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/667223ca-ffbc-4182-beb6-67e68116171f/volumes/kubernetes.io~projected/kube-api-access-t9rtk:{mountpoint:/var/lib/kubelet/pods/667223ca-ffbc-4182-beb6-67e68116171f/volumes/kubernetes.io~projected/kube-api-access-t9rtk major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/689b30e8-936e-49a4-a775-4ce908ad1ab8/volumes/kubernetes.io~projected/kube-api-access-h9pbs:{mountpoint:/var/lib/kubelet/pods/689b30e8-936e-49a4-a775-4ce908ad1ab8/volumes/kubernetes.io~projected/kube-api-access-h9pbs major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/689b30e8-936e-49a4-a775-4ce908ad1ab8/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/689b30e8-936e-49a4-a775-4ce908ad1ab8/volumes/kubernetes.io~secret/serving-cert major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6aecdeeb-4efb-413e-a26c-9270468062a8/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/6aecdeeb-4efb-413e-a26c-9270468062a8/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6cb79353-f35d-4dde-ad87-f0d03352cc49/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/6cb79353-f35d-4dde-ad87-f0d03352cc49/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6cb79353-f35d-4dde-ad87-f0d03352cc49/volumes/kubernetes.io~projected/kube-api-access-7rwgq:{mountpoint:/var/lib/kubelet/pods/6cb79353-f35d-4dde-ad87-f0d03352cc49/volumes/kubernetes.io~projected/kube-api-access-7rwgq major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6cb79353-f35d-4dde-ad87-f0d03352cc49/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/6cb79353-f35d-4dde-ad87-f0d03352cc49/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/735e8903-5dfe-4cb2-8937-cfc3d09ffa36/volumes/kubernetes.io~projected/kube-api-access-r6w5f:{mountpoint:/var/lib/kubelet/pods/735e8903-5dfe-4cb2-8937-cfc3d09ffa36/volumes/kubernetes.io~projected/kube-api-access-r6w5f major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/735e8903-5dfe-4cb2-8937-cfc3d09ffa36/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/735e8903-5dfe-4cb2-8937-cfc3d09ffa36/volumes/kubernetes.io~secret/cert major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c819325-a41e-445e-858d-fa52911383fe/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/7c819325-a41e-445e-858d-fa52911383fe/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c819325-a41e-445e-858d-fa52911383fe/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/7c819325-a41e-445e-858d-fa52911383fe/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c819325-a41e-445e-858d-fa52911383fe/volumes/kubernetes.io~projected/kube-api-access-52tkm:{mountpoint:/var/lib/kubelet/pods/7c819325-a41e-445e-858d-fa52911383fe/volumes/kubernetes.io~projected/kube-api-access-52tkm major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145/volumes/kubernetes.io~projected/kube-api-access-dkvnm:{mountpoint:/var/lib/kubelet/pods/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145/volumes/kubernetes.io~projected/kube-api-access-dkvnm major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86d13ddf-e3f4-49ad-a779-b9590f50076d/volumes/kubernetes.io~projected/kube-api-access-6bnks:{mountpoint:/var/lib/kubelet/pods/86d13ddf-e3f4-49ad-a779-b9590f50076d/volumes/kubernetes.io~projected/kube-api-access-6bnks major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/908118ed-04c5-483d-bb93-61b5d1116259/volumes/kubernetes.io~projected/kube-api-access-wsh4d:{mountpoint:/var/lib/kubelet/pods/908118ed-04c5-483d-bb93-61b5d1116259/volumes/kubernetes.io~projected/kube-api-access-wsh4d major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/913d1e5a-3dfc-4282-b5f0-95c4484b31da/volumes/kubernetes.io~projected/kube-api-access-2wm66:{mountpoint:/var/lib/kubelet/pods/913d1e5a-3dfc-4282-b5f0-95c4484b31da/volumes/kubernetes.io~projected/kube-api-access-2wm66 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2/volumes/kubernetes.io~projected/kube-api-access-rgqk7:{mountpoint:/var/lib/kubelet/pods/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2/volumes/kubernetes.io~projected/kube-api-access-rgqk7 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2/volumes/kubernetes.io~secret/serving-cert major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a173095c-7c94-4533-8529-81b464886f05/volumes/kubernetes.io~projected/kube-api-access-8jr76:{mountpoint:/var/lib/kubelet/pods/a173095c-7c94-4533-8529-81b464886f05/volumes/kubernetes.io~projected/kube-api-access-8jr76 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8dde726-eabf-405e-9d38-a110274353ca/volumes/kubernetes.io~projected/kube-api-access-jnl7p:{mountpoint:/var/lib/kubelet/pods/a8dde726-eabf-405e-9d38-a110274353ca/volumes/kubernetes.io~projected/kube-api-access-jnl7p major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1de1047-c4c9-4d0b-8f72-bafc79987c2e/volumes/kubernetes.io~projected/kube-api-access-kw7ww:{mountpoint:/var/lib/kubelet/pods/c1de1047-c4c9-4d0b-8f72-bafc79987c2e/volumes/kubernetes.io~projected/kube-api-access-kw7ww major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1de1047-c4c9-4d0b-8f72-bafc79987c2e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c1de1047-c4c9-4d0b-8f72-bafc79987c2e/volumes/kubernetes.io~secret/serving-cert major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d683af01-29f3-4634-ae6d-3de8a5c4fd85/volumes/kubernetes.io~projected/kube-api-access-k7dvp:{mountpoint:/var/lib/kubelet/pods/d683af01-29f3-4634-ae6d-3de8a5c4fd85/volumes/kubernetes.io~projected/kube-api-access-k7dvp major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8606abd-d458-4bee-b150-461becfbb83b/volumes/kubernetes.io~projected/kube-api-access-4kgwx:{mountpoint:/var/lib/kubelet/pods/d8606abd-d458-4bee-b150-461becfbb83b/volumes/kubernetes.io~projected/kube-api-access-4kgwx major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/daa3da9a-3a5f-49a4-8a09-0036fdc5aa44/volumes/kubernetes.io~projected/kube-api-access-kscjb:{mountpoint:/var/lib/kubelet/pods/daa3da9a-3a5f-49a4-8a09-0036fdc5aa44/volumes/kubernetes.io~projected/kube-api-access-kscjb major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbea6432-5ba3-49f3-8a83-db2605a3aa2c/volumes/kubernetes.io~projected/kube-api-access-cfdhj:{mountpoint:/var/lib/kubelet/pods/dbea6432-5ba3-49f3-8a83-db2605a3aa2c/volumes/kubernetes.io~projected/kube-api-access-cfdhj major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3/volumes/kubernetes.io~projected/kube-api-access-s2r5h:{mountpoint:/var/lib/kubelet/pods/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3/volumes/kubernetes.io~projected/kube-api-access-s2r5h major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eddf03ef-53ea-4fc0-9f3f-da7d741cf06a/volumes/kubernetes.io~projected/kube-api-access-wr45n:{mountpoint:/var/lib/kubelet/pods/eddf03ef-53ea-4fc0-9f3f-da7d741cf06a/volumes/kubernetes.io~projected/kube-api-access-wr45n major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee7974f8-92b4-498a-bf54-2c3a353f4b49/volumes/kubernetes.io~projected/kube-api-access-6jkxn:{mountpoint:/var/lib/kubelet/pods/ee7974f8-92b4-498a-bf54-2c3a353f4b49/volumes/kubernetes.io~projected/kube-api-access-6jkxn major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffb2a03c-414c-4d68-ab86-212c554270e2/volumes/kubernetes.io~projected/kube-api-access-n2tzh:{mountpoint:/var/lib/kubelet/pods/ffb2a03c-414c-4d68-ab86-212c554270e2/volumes/kubernetes.io~projected/kube-api-access-n2tzh major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/9f82ab46ea486a3eb1b8e5599c4010bb5ffd99c3f655bde487db604386f5857a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/57e39d28d00e8148360acdf3ca2fbd9b7178ef018b7677235287c860df1f751a/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/716cbf5781bad83d63d0d95f225b2b46a66ccc04291a753ace438a7b84cf4729/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/8c1653577f3932fee36564d6388e13adaf595615252ecb731347d5ca6c3de052/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/19f8e6fed5dcba8139593583b15646561ee6a8373a41549aad2608e4076facba/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/8aeef91ebe71b418f4d6ba8434827e7b5c6c71bccd4d2a45290ea1372dbaca0b/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/bbdfd71ba14df756b383a27ef1b84050c8f7223adf2d0d7050071c9dc9d4f2e3/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/eee12342cb147654f148e7d5cd66c79095563f41f8bc00fb4d70647eaa32586f/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/0215a3ee65eaf2a71de4a204268536271855430f6fc11430b073bc8e002639d9/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/52ee3b44996233065471ae58cc5ca5c109b741cd6d72754847e2848369fc8aa6/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/322cc6d8316d8360059888a740d47b75e75a0672f60451eeea93003cffc3f21e/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/9a085c753bd1281043ba8c1e2b3a9905d77de95bf2eaaf7f8d64eed290a4356a/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/5308fc8c1bd7c1549e33fe9dd1b1698d60c8698cf6512ec39b998fd431fcd2db/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/b556d8eb4576fb643ab1d0bc2956dcb7a41887175430afbd2f2fc59ceae3cb6a/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/efdf1da1a52b93f3ca28b91d7363cf300ccdd9e75a57e9f0e53992f812fc3af7/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/4894c4951298b58cd671464c36dc5768c304bf335ae601439ae8b88ead96a256/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/e6935606b868f11a53606c944e2864b6bc90338dfb1f6beb1137ca85e43d5e7c/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/f742f1964efeddb5ad7f416aa95154a81d1c719af9900346b78e094f0610e049/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/07a290fb200fc823ba13056b452f9f0af0831c6956b0b14ecc0f16ee7cbb9050/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/30bf5f03ed767a0327357d993c03ee53b55be0d6f073e6e1a76389165b26d3a7/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/75ccd783f43f862188dca7b00ee75d651503f7fec2c160c9a1ea87801d1693f7/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/bdcd1d73db791524bb835a5eb3d59569ff66f6aea63ed73fa86e95d386f2a7dc/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/d2727294b40d276cdeea71ea4101d10c896eeb47973360111731666684153074/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/0c1c932be6445424e63d3af02a9b96e036d495b705b201c5b572e1d80b8b143e/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/030721aad2b34a56976389c39dd098ca227ebf30721184153524ecf5aed3b836/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/ad1cdee62813d8287be932fe3bf8ba6bdb1da8059303089c21b54ef6fdc6b15d/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/78d3c873066c7a47dd3fa3108055c37dacf84deb3a5fa6869430738a76e7c9ad/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/99f3cfe3fb71062412b0d878162c8f18cd57a37fdcab2cf11f4450da38b2778a/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/fa43700bb9b5bccca393a9a1260c883e27c7a555d24d9aefe25d8c54581e3ab8/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/b4850ec638dfbabd9389b038149582f613eb7d9cc36dd2cfce73d97b64862926/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/2502be4732de5e46455c74dfed9a078614a0fae194b429a412c6223bcd162a6d/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/a11694286b91a42118fdea4ac8fea1aa9093c26b14d710acce945d49910d28a6/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/8db984e3071c57efbce25aac7ebc106b60d6cebdcb6eea633ded75c3352e4831/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/7f1c2e8683c3683d02814cc36db297b3f6c15c179304cc506bf0d6509ea9844f/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/15cd48296ebd6a5001f1af0811b54cb633773dda69469e352155afd3e47f4a59/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/1a67b1acef8e4692d9ddf82e6dc572b8b4c9d740b4a3facb74ab1da0d1b4554d/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/3e2c7464ea85e53867c8f9153b26b3d675f142a36f8445d8f8ae1c46701e1e1e/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/2ba5f293df187fb62678523497bb8be3c41a83844c7b016c7f549694da1275ba/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/915bdb36dfc05ad57de0692ed9a4fef1ad9d724054bf74717bdcfa3822d11b14/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/a86bb01677a67536aea893d32bdc2fe437678c88c2d2466b40f74cb5579e2214/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/c39ce30b025af1fbc61cde8f83d95d84075c794b9e6bd5ad5906d958bb153114/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/01403334d10c75ea32d06bd5d418a503e394f88fd5beb2ae59ff7b8abefe551b/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/8fa1ca1c3d3887ac196348eaa766110a7d9cd7a8b71951d11ece52f3a8af2865/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/09d946fecd02e8d9b8e367db130d6aba57035b83ecf5d13db7fb70a5d0f77599/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/663a24e15ebd76805cbbd2c02ac4f5bd5f84424b5eb4d14e2ee0fecbc693548f/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/9f37f574de0eac804af27e4a51a8b776ce29a70244ceef2e0a5e954742867ae0/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/92245f97f99e5d3fd7cdfa33e466b3ae078f2c6476a92209f0cf0f64a3ef396c/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/f69ac971886b7dcb4de9a9a5aafa7e54cf02b9ea52f93ef23b8fcd2176ec6e5d/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/42e028a5dda33084fc8bddbc818128f26a61e82c1c6eec1f54e12d00f5525721/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/126557e952d8753b7973af7438c3c700c1a2b623daebb65c447f91bd1d7f930b/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/8b26b8df09abe2ec98c3ba01c4bd212040fe727a6a10b3cbb2bdeab729c35c77/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/7f9e3a49e68fa03ca4e3a2d2a4be82fc48e3c7ae6db4f257f1b48c7bc666319a/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/ab5a0b5037f4a2597855a83d34f9ec46501a5d62dfdfa9d719f30719fc8dead7/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/3cbf6443f68e4ee78c743b4de242bc580871584a638c410f90f9dc09d3a48d3a/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/e18772aa9f5eb0a505a3639551c2c46f081e725367fee121b51a551f064de8ff/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/f4fc00420baceffcf36309316e4fc9a6a9153e98ae24c4dcb4a9fd7b7e876b34/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/6eb56eb721d3906b3fb446e9af9b13897375e9a930b36a8452910e77ee37ac40/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/8cde1e5a2c5d2eeb2d8472f7f5d036e900caa87e5704c85f1e77352a126d47be/merged major:0 minor:97 fsType:overlay blockSize:0}]
Nov 03 00:05:17.185271 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.184932    5012 manager.go:217] Machine: {Timestamp:2026-11-03 00:05:17.184459131 +0000 UTC m=+0.064043165 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100199 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23955d65bf5fb8d06ae7fd50c2ea74 SystemUUID:ec23955d-65bf-5fb8-d06a-e7fd50c2ea74 BootID:85306522-6d3a-44ee-84ad-5fbca3969858 Filesystems:[{Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/015c1f9d-d4c4-4749-978a-3cd2166c7bf8/volumes/kubernetes.io~projected/kube-api-access-f59nd DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12eb002e-9765-4443-b488-6cec02b9c961/volumes/kubernetes.io~projected/kube-api-access-5xcfc DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd42c6e0fad9c6df6a09e53e66c81812205707ec597690d24a3739ea61411c03/userdata/shm DeviceMajor:0 DeviceMinor:305 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7c819325-a41e-445e-858d-fa52911383fe/volumes/kubernetes.io~projected/kube-api-access-52tkm DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/38b66660-5608-4e82-a1eb-db3d6ce57ca5/volumes/kubernetes.io~projected/kube-api-access-chv2w DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6aecdeeb-4efb-413e-a26c-9270468062a8/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:299 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a22d53dec8b80ea7bce25b6229cff9d1c7b321b62c8adfb26174dc5682256fa4/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eddf03ef-53ea-4fc0-9f3f-da7d741cf06a/volumes/kubernetes.io~projected/kube-api-access-wr45n DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32925b0fda6815092f415eee6ae94fde3490062f322d962538f8ff5fcf9c953a/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2/volumes/kubernetes.io~projected/kube-api-access-rgqk7 DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/28e0cef7-ba0d-4546-a193-2f411c441350/volumes/kubernetes.io~projected/kube-api-access-rmwfj DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ffb3ce37f4ea01beca43eaf7f71968d3f6bf22a280aa4e6de499ade77a4ab42/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4d7b91ff-203c-4e03-9ad7-9b60215b61b6/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:241 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e601c2fff52128b9a39f52622375e33e465c0b247a12cfec41cc65c556022d1e/userdata/shm DeviceMajor:0 DeviceMinor:249 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/689b30e8-936e-49a4-a775-4ce908ad1ab8/volumes/kubernetes.io~projected/kube-api-access-h9pbs DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25b8cd2ec77476e68d59d598da6f50bc0901483679ddc07512611488c8c5469e/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7c819325-a41e-445e-858d-fa52911383fe/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f3628bbb595515569d3baf7515603db2ec1f972bfaf52e530cda709833793d2/userdata/shm DeviceMajor:0 DeviceMinor:247 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4c5d55cb-0306-4dc1-b25b-c2444f148af6/volumes/kubernetes.io~projected/kube-api-access-x7b5n DeviceMajor:0 DeviceMinor:322 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6cb79353-f35d-4dde-ad87-f0d03352cc49/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33492d80a3afb785c34967b66d6b9022171cd7cc706d7549e07017fca5bca917/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7dbc28b60bb8f7ec99da45a1dfa4c162f3c620de1a64f8d9e6c094405e4ee516/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a173095c-7c94-4533-8529-81b464886f05/volumes/kubernetes.io~projected/kube-api-access-8jr76 DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/47ea3616-f54d-4603-a56f-5c549d66eb81/volumes/kubernetes.io~projected/kube-api-access-255z4 DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/47258bdf-947a-4a28-86f4-02871f64941a/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84cd2d9a687e85e59dd66448d9c7f03e7d89bf34dab38621ced8b345f589eeb0/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/735e8903-5dfe-4cb2-8937-cfc3d09ffa36/volumes/kubernetes.io~projected/kube-api-access-r6w5f DeviceMajor:0 DeviceMinor:242 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ce6a3437092f12bae141073692ecf67563bf2879195884d2fabe043160a95ec/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/913d1e5a-3dfc-4282-b5f0-95c4484b31da/volumes/kubernetes.io~projected/kube-api-access-2wm66 DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffb2a03c-414c-4d68-ab86-212c554270e2/volumes/kubernetes.io~projected/kube-api-access-n2tzh DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/667223ca-ffbc-4182-beb6-67e68116171f/volumes/kubernetes.io~projected/kube-api-access-t9rtk DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a8dde726-eabf-405e-9d38-a110274353ca/volumes/kubernetes.io~projected/kube-api-access-jnl7p DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e48db05aa0f14e58da60aba11b7801ff25fac95d8b6797d39edd78abdf2b4ee/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6cb79353-f35d-4dde-ad87-f0d03352cc49/volumes/kubernetes.io~projected/kube-api-access-7rwgq DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/689b30e8-936e-49a4-a775-4ce908ad1ab8/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c819325-a41e-445e-858d-fa52911383fe/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fe36277fd7b6c5722e4848adb1f4e278fff67e03f4a714cb3d478fb71d58794c/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4d7b91ff-203c-4e03-9ad7-9b60215b61b6/volumes/kubernetes.io~projected/kube-api-access-bbzll DeviceMajor:0 DeviceMinor:240 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12eb002e-9765-4443-b488-6cec02b9c961/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/064e4777f893298302176dc608810e2c15662c38066e2bacbb86c206a1fdfa35/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fdaa2ff297d604e60bdd287263f647e7383635a2e8c3eccef5fbd2e80e83b271/userdata/shm DeviceMajor:0 DeviceMinor:327 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ddf0b02c07944ca9b610fb313cc5817c926a312bebea337199d8e15ec25e4276/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d683af01-29f3-4634-ae6d-3de8a5c4fd85/volumes/kubernetes.io~projected/kube-api-access-k7dvp DeviceMajor:0 DeviceMinor:304 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/86d13ddf-e3f4-49ad-a779-b9590f50076d/volumes/kubernetes.io~projected/kube-api-access-6bnks DeviceMajor:0 DeviceMinor:315 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb6385809f85d1b7277cd035d86d0eb1b4ed29b00d661d6406ffe3bcf1d7ae20/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e534f8d6a73d1ff175ee0a5a2695bd877777cb30a483c171cafda7c5946ace57/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d310053e8d88c0ce69bc8a9f7d169268a0d1b263840568fe264b8105f1815dce/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12eb002e-9765-4443-b488-6cec02b9c961/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3354abe343c02c13547e14d64d5169f536b3fa0bc46cfd92bd9b306ed7d40a5/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e98a161f0c761579d58a105c20911bdaae023367f88ffc6be5f4dafd85a3fa77/userdata/shm DeviceMajor:0 DeviceMinor:307 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dce57bb4de821416aa2640454e1413069f4756379b69eb399c55299f4d1d1219/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4d7b91ff-203c-4e03-9ad7-9b60215b61b6/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d7b91ff-203c-4e03-9ad7-9b60215b61b6/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6cb79353-f35d-4dde-ad87-f0d03352cc49/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/38b66660-5608-4e82-a1eb-db3d6ce57ca5/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dbea6432-5ba3-49f3-8a83-db2605a3aa2c/volumes/kubernetes.io~projected/kube-api-access-cfdhj DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/735e8903-5dfe-4cb2-8937-cfc3d09ffa36/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:234 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/908118ed-04c5-483d-bb93-61b5d1116259/volumes/kubernetes.io~projected/kube-api-access-wsh4d DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/61d5adce-3738-4cad-aace-39d9d33a739a/volumes/kubernetes.io~projected/kube-api-access-chvlq DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c1de1047-c4c9-4d0b-8f72-bafc79987c2e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0e33ec8758b4860c3c56bb4804adbb0957f25001607aa4efca11dcb808febb3/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f91b5735012147ec87a0fee6fe44f7791385e421c0d95eb9e70b9d5d8ea000f2/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/daa3da9a-3a5f-49a4-8a09-0036fdc5aa44/volumes/kubernetes.io~projected/kube-api-access-kscjb DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ee7974f8-92b4-498a-bf54-2c3a353f4b49/volumes/kubernetes.io~projected/kube-api-access-6jkxn DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3/volumes/kubernetes.io~projected/kube-api-access-s2r5h DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d8606abd-d458-4bee-b150-461becfbb83b/volumes/kubernetes.io~projected/kube-api-access-4kgwx DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c1de1047-c4c9-4d0b-8f72-bafc79987c2e/volumes/kubernetes.io~projected/kube-api-access-kw7ww DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145/volumes/kubernetes.io~projected/kube-api-access-dkvnm DeviceMajor:0 DeviceMinor:239 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:064e4777f893298 MacAddress:1e:10:b4:15:8b:2f Speed:10000 Mtu:8901} {Name:33492d80a3afb78 MacAddress:fa:88:61:72:a9:74 Speed:10000 Mtu:8901} {Name:3e48db05aa0f14e MacAddress:4a:d2:15:45:9c:18 Speed:10000 Mtu:8901} {Name:3f3628bbb595515 MacAddress:46:b8:68:e6:52:dc Speed:10000 Mtu:8901} {Name:7dbc28b60bb8f7e MacAddress:86:b5:fb:41:67:e3 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f2:20:06:83 Speed:0 Mtu:9001} {Name:br-int MacAddress:2e:96:14:53:65:8e Speed:0 Mtu:8901} {Name:d0e33ec8758b486 MacAddress:e2:db:3e:72:9a:20 Speed:10000 Mtu:8901} {Name:d310053e8d88c0c MacAddress:ea:cf:11:82:b5:45 Speed:10000 Mtu:8901} {Name:dd42c6e0fad9c6d MacAddress:fe:86:97:c3:50:49 Speed:10000 Mtu:8901} {Name:e3354abe343c02c MacAddress:2a:e4:a1:fc:41:42 Speed:10000 Mtu:8901} {Name:e534f8d6a73d1ff MacAddress:ba:4c:0a:e7:d2:6d Speed:10000 Mtu:8901} {Name:e601c2fff52128b MacAddress:56:92:97:05:f8:0a Speed:10000 Mtu:8901} {Name:e98a161f0c76157 MacAddress:8e:46:69:00:6a:9b Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f2:20:06:83 Speed:0 Mtu:9001} {Name:fdaa2ff297d604e MacAddress:6a:86:ee:51:ea:3a Speed:10000 Mtu:8901} {Name:fe36277fd7b6c57 MacAddress:a2:24:6d:17:62:03 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:1a:61:4e:7c:ca:67 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:a6:62:70:c2:11:83 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 00:05:17.185475 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.185276    5012 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 00:05:17.185475 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.185324    5012 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 00:05:17.185628 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.185607    5012 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 00:05:17.185743 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.185629    5012 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-161.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 00:05:17.185787 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.185760    5012 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 00:05:17.185787 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.185768    5012 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 00:05:17.185787 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.185781    5012 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 00:05:17.185865 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.185797    5012 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 00:05:17.185865 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.185815    5012 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 00:05:17.185998 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.185988    5012 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 00:05:17.186090 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.186082    5012 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 00:05:17.186119 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.186116    5012 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 00:05:17.186146 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.186124    5012 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 00:05:17.186146 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.186133    5012 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 00:05:17.186146 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.186143    5012 kubelet.go:405] "Adding apiserver pod source"
Nov 03 00:05:17.186231 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.186150    5012 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 00:05:17.186820 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.186807    5012 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 00:05:17.186820 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.186819    5012 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 00:05:17.187313 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.187292    5012 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 00:05:17.190381 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.190366    5012 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 00:05:17.190381 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.190380    5012 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 00:05:17.190491 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.190482    5012 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 00:05:17.190663 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.190652    5012 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 00:05:17.190698 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.190681    5012 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 00:05:17.190788 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.190774    5012 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 00:05:17.190788 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.190790    5012 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 00:05:17.190890 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.190796    5012 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 00:05:17.190890 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.190802    5012 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 00:05:17.190890 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.190809    5012 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 00:05:17.190890 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.190818    5012 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 00:05:17.190890 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.190827    5012 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 00:05:17.190890 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.190833    5012 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 00:05:17.190890 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.190839    5012 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 00:05:17.190890 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.190855    5012 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 00:05:17.190890 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.190871    5012 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 00:05:17.190890 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.190879    5012 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 00:05:17.191338 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.190904    5012 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 00:05:17.191338 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.190933    5012 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 00:05:17.194373 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.194359    5012 server.go:1263] "Started kubelet"
Nov 03 00:05:17.194471 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.194447    5012 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 00:05:17.194763 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.194516    5012 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 00:05:17.194844 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.194771    5012 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 00:05:17.194888 ip-10-0-133-161 systemd[1]: Started Kubernetes Kubelet.
Nov 03 00:05:17.195087 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.195071    5012 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 00:05:17.198148 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.198117    5012 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 00:05:17.201944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.201408    5012 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 00:05:17.201944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.201415    5012 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 00:05:17.201944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.201479    5012 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 23:59:37 +0000 UTC" deadline="2028-03-31 18:44:39.193368542 +0000 UTC"
Nov 03 00:05:17.201944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.201497    5012 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12354h39m21.991874766s"
Nov 03 00:05:17.201944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.201538    5012 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 00:05:17.201944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.201545    5012 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 00:05:17.201944 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:17.201619    5012 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-161.ec2.internal\" not found"
Nov 03 00:05:17.201944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.201821    5012 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 00:05:17.204021 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.203962    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8dde726-eabf-405e-9d38-a110274353ca" volumeName="kubernetes.io/configmap/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-config" seLinuxMountContext=""
Nov 03 00:05:17.204021 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.203988    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38b66660-5608-4e82-a1eb-db3d6ce57ca5" volumeName="kubernetes.io/projected/38b66660-5608-4e82-a1eb-db3d6ce57ca5-kube-api-access-chv2w" seLinuxMountContext=""
Nov 03 00:05:17.204021 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.203997    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47258bdf-947a-4a28-86f4-02871f64941a" volumeName="kubernetes.io/configmap/47258bdf-947a-4a28-86f4-02871f64941a-konnectivity-ca" seLinuxMountContext=""
Nov 03 00:05:17.204021 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204006    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0d4ed33-0500-4c65-ba72-b2a584b6bdf2" volumeName="kubernetes.io/projected/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-kube-api-access-rgqk7" seLinuxMountContext=""
Nov 03 00:05:17.204021 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204013    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1de1047-c4c9-4d0b-8f72-bafc79987c2e" volumeName="kubernetes.io/projected/c1de1047-c4c9-4d0b-8f72-bafc79987c2e-kube-api-access-kw7ww" seLinuxMountContext=""
Nov 03 00:05:17.204021 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204021    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="015c1f9d-d4c4-4749-978a-3cd2166c7bf8" volumeName="kubernetes.io/configmap/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-multus-daemon-config" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204028    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cb79353-f35d-4dde-ad87-f0d03352cc49" volumeName="kubernetes.io/projected/6cb79353-f35d-4dde-ad87-f0d03352cc49-kube-api-access-7rwgq" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204036    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c819325-a41e-445e-858d-fa52911383fe" volumeName="kubernetes.io/empty-dir/7c819325-a41e-445e-858d-fa52911383fe-tmp" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204045    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86d13ddf-e3f4-49ad-a779-b9590f50076d" volumeName="kubernetes.io/projected/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-api-access-6bnks" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204052    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="015c1f9d-d4c4-4749-978a-3cd2166c7bf8" volumeName="kubernetes.io/configmap/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-cni-binary-copy" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204059    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28e0cef7-ba0d-4546-a193-2f411c441350" volumeName="kubernetes.io/projected/28e0cef7-ba0d-4546-a193-2f411c441350-kube-api-access-rmwfj" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204066    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c5d55cb-0306-4dc1-b25b-c2444f148af6" volumeName="kubernetes.io/projected/4c5d55cb-0306-4dc1-b25b-c2444f148af6-kube-api-access-x7b5n" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204073    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="908118ed-04c5-483d-bb93-61b5d1116259" volumeName="kubernetes.io/projected/908118ed-04c5-483d-bb93-61b5d1116259-kube-api-access-wsh4d" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204081    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1de1047-c4c9-4d0b-8f72-bafc79987c2e" volumeName="kubernetes.io/secret/c1de1047-c4c9-4d0b-8f72-bafc79987c2e-serving-cert" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204090    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d683af01-29f3-4634-ae6d-3de8a5c4fd85" volumeName="kubernetes.io/projected/d683af01-29f3-4634-ae6d-3de8a5c4fd85-kube-api-access-k7dvp" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204098    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3" volumeName="kubernetes.io/configmap/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-trusted-ca" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204118    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12eb002e-9765-4443-b488-6cec02b9c961" volumeName="kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-default-certificate" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204133    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47ea3616-f54d-4603-a56f-5c549d66eb81" volumeName="kubernetes.io/configmap/47ea3616-f54d-4603-a56f-5c549d66eb81-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204145    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47ea3616-f54d-4603-a56f-5c549d66eb81" volumeName="kubernetes.io/projected/47ea3616-f54d-4603-a56f-5c549d66eb81-kube-api-access-255z4" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204154    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d7b91ff-203c-4e03-9ad7-9b60215b61b6" volumeName="kubernetes.io/secret/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-image-registry-private-configuration" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204161    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="689b30e8-936e-49a4-a775-4ce908ad1ab8" volumeName="kubernetes.io/secret/689b30e8-936e-49a4-a775-4ce908ad1ab8-serving-cert" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204168    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cb79353-f35d-4dde-ad87-f0d03352cc49" volumeName="kubernetes.io/configmap/6cb79353-f35d-4dde-ad87-f0d03352cc49-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204175    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cb79353-f35d-4dde-ad87-f0d03352cc49" volumeName="kubernetes.io/secret/6cb79353-f35d-4dde-ad87-f0d03352cc49-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204182    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cdb4c55-92a6-4791-8ab8-fcb22ef3a145" volumeName="kubernetes.io/empty-dir/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-tmp-dir" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204189    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12eb002e-9765-4443-b488-6cec02b9c961" volumeName="kubernetes.io/projected/12eb002e-9765-4443-b488-6cec02b9c961-kube-api-access-5xcfc" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204199    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cb79353-f35d-4dde-ad87-f0d03352cc49" volumeName="kubernetes.io/configmap/6cb79353-f35d-4dde-ad87-f0d03352cc49-ovnkube-config" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204205    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="735e8903-5dfe-4cb2-8937-cfc3d09ffa36" volumeName="kubernetes.io/projected/735e8903-5dfe-4cb2-8937-cfc3d09ffa36-kube-api-access-r6w5f" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204212    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cdb4c55-92a6-4791-8ab8-fcb22ef3a145" volumeName="kubernetes.io/projected/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-kube-api-access-dkvnm" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204235    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0d4ed33-0500-4c65-ba72-b2a584b6bdf2" volumeName="kubernetes.io/configmap/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204245    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1de1047-c4c9-4d0b-8f72-bafc79987c2e" volumeName="kubernetes.io/configmap/c1de1047-c4c9-4d0b-8f72-bafc79987c2e-trusted-ca" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204252    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbea6432-5ba3-49f3-8a83-db2605a3aa2c" volumeName="kubernetes.io/configmap/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-telemetry-config" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204261    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c819325-a41e-445e-858d-fa52911383fe" volumeName="kubernetes.io/empty-dir/7c819325-a41e-445e-858d-fa52911383fe-etc-tuned" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204268    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0d4ed33-0500-4c65-ba72-b2a584b6bdf2" volumeName="kubernetes.io/empty-dir/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-tmp" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204275    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0d4ed33-0500-4c65-ba72-b2a584b6bdf2" volumeName="kubernetes.io/secret/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-serving-cert" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204282    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8dde726-eabf-405e-9d38-a110274353ca" volumeName="kubernetes.io/projected/a8dde726-eabf-405e-9d38-a110274353ca-kube-api-access-jnl7p" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204288    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="daa3da9a-3a5f-49a4-8a09-0036fdc5aa44" volumeName="kubernetes.io/configmap/daa3da9a-3a5f-49a4-8a09-0036fdc5aa44-serviceca" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204295    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3" volumeName="kubernetes.io/empty-dir/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-ca-trust-extracted" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204303    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61d5adce-3738-4cad-aace-39d9d33a739a" volumeName="kubernetes.io/projected/61d5adce-3738-4cad-aace-39d9d33a739a-kube-api-access-chvlq" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204310    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="908118ed-04c5-483d-bb93-61b5d1116259" volumeName="kubernetes.io/configmap/908118ed-04c5-483d-bb93-61b5d1116259-iptables-alerter-script" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204319    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbea6432-5ba3-49f3-8a83-db2605a3aa2c" volumeName="kubernetes.io/projected/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-kube-api-access-cfdhj" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204330    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee7974f8-92b4-498a-bf54-2c3a353f4b49" volumeName="kubernetes.io/projected/ee7974f8-92b4-498a-bf54-2c3a353f4b49-kube-api-access-6jkxn" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204341    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb2a03c-414c-4d68-ab86-212c554270e2" volumeName="kubernetes.io/projected/ffb2a03c-414c-4d68-ab86-212c554270e2-kube-api-access-n2tzh" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204349    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47ea3616-f54d-4603-a56f-5c549d66eb81" volumeName="kubernetes.io/configmap/47ea3616-f54d-4603-a56f-5c549d66eb81-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204358    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d7b91ff-203c-4e03-9ad7-9b60215b61b6" volumeName="kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-bound-sa-token" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204369    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="913d1e5a-3dfc-4282-b5f0-95c4484b31da" volumeName="kubernetes.io/projected/913d1e5a-3dfc-4282-b5f0-95c4484b31da-kube-api-access-2wm66" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204377    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a173095c-7c94-4533-8529-81b464886f05" volumeName="kubernetes.io/projected/a173095c-7c94-4533-8529-81b464886f05-kube-api-access-8jr76" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204383    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb2a03c-414c-4d68-ab86-212c554270e2" volumeName="kubernetes.io/configmap/ffb2a03c-414c-4d68-ab86-212c554270e2-serviceca" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204391    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38b66660-5608-4e82-a1eb-db3d6ce57ca5" volumeName="kubernetes.io/configmap/38b66660-5608-4e82-a1eb-db3d6ce57ca5-config" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204398    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38b66660-5608-4e82-a1eb-db3d6ce57ca5" volumeName="kubernetes.io/secret/38b66660-5608-4e82-a1eb-db3d6ce57ca5-serving-cert" seLinuxMountContext=""
Nov 03 00:05:17.204378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204406    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3" volumeName="kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-bound-sa-token" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204413    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3" volumeName="kubernetes.io/secret/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-installation-pull-secrets" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204421    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3" volumeName="kubernetes.io/secret/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-image-registry-private-configuration" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204429    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="015c1f9d-d4c4-4749-978a-3cd2166c7bf8" volumeName="kubernetes.io/projected/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-kube-api-access-f59nd" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204436    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47258bdf-947a-4a28-86f4-02871f64941a" volumeName="kubernetes.io/secret/47258bdf-947a-4a28-86f4-02871f64941a-agent-certs" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204442    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47ea3616-f54d-4603-a56f-5c549d66eb81" volumeName="kubernetes.io/configmap/47ea3616-f54d-4603-a56f-5c549d66eb81-cni-binary-copy" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204451    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d7b91ff-203c-4e03-9ad7-9b60215b61b6" volumeName="kubernetes.io/configmap/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-certificates" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204463    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28e0cef7-ba0d-4546-a193-2f411c441350" volumeName="kubernetes.io/empty-dir/28e0cef7-ba0d-4546-a193-2f411c441350-tmp-dir" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204470    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d7b91ff-203c-4e03-9ad7-9b60215b61b6" volumeName="kubernetes.io/configmap/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-trusted-ca" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204477    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d7b91ff-203c-4e03-9ad7-9b60215b61b6" volumeName="kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-kube-api-access-bbzll" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204487    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="689b30e8-936e-49a4-a775-4ce908ad1ab8" volumeName="kubernetes.io/projected/689b30e8-936e-49a4-a775-4ce908ad1ab8-kube-api-access-h9pbs" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204494    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c819325-a41e-445e-858d-fa52911383fe" volumeName="kubernetes.io/projected/7c819325-a41e-445e-858d-fa52911383fe-kube-api-access-52tkm" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204502    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1de1047-c4c9-4d0b-8f72-bafc79987c2e" volumeName="kubernetes.io/configmap/c1de1047-c4c9-4d0b-8f72-bafc79987c2e-config" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204508    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3" volumeName="kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-kube-api-access-s2r5h" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204515    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12eb002e-9765-4443-b488-6cec02b9c961" volumeName="kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-stats-auth" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204522    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="689b30e8-936e-49a4-a775-4ce908ad1ab8" volumeName="kubernetes.io/configmap/689b30e8-936e-49a4-a775-4ce908ad1ab8-config" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204529    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86d13ddf-e3f4-49ad-a779-b9590f50076d" volumeName="kubernetes.io/empty-dir/86d13ddf-e3f4-49ad-a779-b9590f50076d-data-volume" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204536    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0d4ed33-0500-4c65-ba72-b2a584b6bdf2" volumeName="kubernetes.io/empty-dir/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-snapshots" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204543    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8606abd-d458-4bee-b150-461becfbb83b" volumeName="kubernetes.io/projected/d8606abd-d458-4bee-b150-461becfbb83b-kube-api-access-4kgwx" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204550    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="daa3da9a-3a5f-49a4-8a09-0036fdc5aa44" volumeName="kubernetes.io/projected/daa3da9a-3a5f-49a4-8a09-0036fdc5aa44-kube-api-access-kscjb" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204557    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eddf03ef-53ea-4fc0-9f3f-da7d741cf06a" volumeName="kubernetes.io/projected/eddf03ef-53ea-4fc0-9f3f-da7d741cf06a-kube-api-access-wr45n" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204564    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="735e8903-5dfe-4cb2-8937-cfc3d09ffa36" volumeName="kubernetes.io/secret/735e8903-5dfe-4cb2-8937-cfc3d09ffa36-cert" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204571    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb5193c7-10f4-4436-9c69-20e6087439c2" volumeName="kubernetes.io/configmap/bb5193c7-10f4-4436-9c69-20e6087439c2-nginx-conf" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204577    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3" volumeName="kubernetes.io/configmap/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-certificates" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204589    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cb79353-f35d-4dde-ad87-f0d03352cc49" volumeName="kubernetes.io/configmap/6cb79353-f35d-4dde-ad87-f0d03352cc49-env-overrides" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204597    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cdb4c55-92a6-4791-8ab8-fcb22ef3a145" volumeName="kubernetes.io/configmap/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-config-volume" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204605    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d7b91ff-203c-4e03-9ad7-9b60215b61b6" volumeName="kubernetes.io/secret/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-installation-pull-secrets" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204613    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aecdeeb-4efb-413e-a26c-9270468062a8" volumeName="kubernetes.io/secret/6aecdeeb-4efb-413e-a26c-9270468062a8-original-pull-secret" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204620    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d7b91ff-203c-4e03-9ad7-9b60215b61b6" volumeName="kubernetes.io/empty-dir/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-ca-trust-extracted" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204627    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="667223ca-ffbc-4182-beb6-67e68116171f" volumeName="kubernetes.io/projected/667223ca-ffbc-4182-beb6-67e68116171f-kube-api-access-t9rtk" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204634    5012 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0d4ed33-0500-4c65-ba72-b2a584b6bdf2" volumeName="kubernetes.io/configmap/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-service-ca-bundle" seLinuxMountContext=""
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204641    5012 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.204646    5012 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.205807    5012 factory.go:55] Registering systemd factory
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.205824    5012 factory.go:223] Registration of the systemd container factory successfully
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.206053    5012 factory.go:153] Registering CRI-O factory
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.206070    5012 factory.go:223] Registration of the crio container factory successfully
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.206122    5012 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.206140    5012 factory.go:103] Registering Raw factory
Nov 03 00:05:17.206274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.206156    5012 manager.go:1199] Started watching for new ooms in manager
Nov 03 00:05:17.207756 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.206494    5012 manager.go:319] Starting recovery of all containers
Nov 03 00:05:17.219692 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.219669    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/1.log"
Nov 03 00:05:17.221396 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.221376    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/1.log"
Nov 03 00:05:17.227240 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.227210    5012 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 00:05:17.228498 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.228470    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:05:17.231220 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.230894    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:05:17.234627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.234609    5012 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 00:05:17.235593 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.235031    5012 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:05:17.235593 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.235453    5012 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 00:05:17.235593 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.235487    5012 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 00:05:17.235593 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.235512    5012 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 00:05:17.235593 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.235540    5012 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:05:17.235593 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:17.235553    5012 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 00:05:17.246066 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.246044    5012 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:05:17.246859 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.246843    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/1.log"
Nov 03 00:05:17.246958 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.246873    5012 generic.go:356] "Generic (PLEG): container finished" podID="c1de1047-c4c9-4d0b-8f72-bafc79987c2e" containerID="82c6612b250a969062f81eafeb8c6730b0ebdea3d3d3cc15571a64b5c5a49b0b" exitCode=255
Nov 03 00:05:17.247888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.247871    5012 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:05:17.256094 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.256072    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:05:17.256417 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.256398    5012 generic.go:356] "Generic (PLEG): container finished" podID="6cb79353-f35d-4dde-ad87-f0d03352cc49" containerID="e38570332cda148b0544675c919794c68ea944b32e69790d0a08eb4daa730a6f" exitCode=1
Nov 03 00:05:17.268552 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.268528    5012 generic.go:356] "Generic (PLEG): container finished" podID="317fa55400daeb49ee1d8f32b909a6db" containerID="2c5b611bbafcad9f4d61ebaad27fd448fe8c66a6ae02abca20f4216e2f97c366" exitCode=0
Nov 03 00:05:17.288366 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.288342    5012 generic.go:356] "Generic (PLEG): container finished" podID="47ea3616-f54d-4603-a56f-5c549d66eb81" containerID="a89a20f6bbb90edb384e916fc2954cf3298fa440d2f6002e1d08c10a14bb3299" exitCode=0
Nov 03 00:05:17.288366 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.288361    5012 generic.go:356] "Generic (PLEG): container finished" podID="47ea3616-f54d-4603-a56f-5c549d66eb81" containerID="417d4b66db938b225546c3b92659b22f8bc814d10295a67151539df47db5e44f" exitCode=0
Nov 03 00:05:17.288366 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.288367    5012 generic.go:356] "Generic (PLEG): container finished" podID="47ea3616-f54d-4603-a56f-5c549d66eb81" containerID="6699297e5cbee148f2ce124ae59f66d58d93aeb4374a6dfbf76dd2760e3175f1" exitCode=0
Nov 03 00:05:17.288572 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.288373    5012 generic.go:356] "Generic (PLEG): container finished" podID="47ea3616-f54d-4603-a56f-5c549d66eb81" containerID="23b1c083551963c71dac3fbdd057ee63396e2db102cb3b6e1a2523ce9853d7e6" exitCode=0
Nov 03 00:05:17.288572 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.288378    5012 generic.go:356] "Generic (PLEG): container finished" podID="47ea3616-f54d-4603-a56f-5c549d66eb81" containerID="05e5ebd187841d1a8498ea1e5ca5bc1d6a9a5c53c454e2b00b84a0f70b5413ed" exitCode=0
Nov 03 00:05:17.288572 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.288384    5012 generic.go:356] "Generic (PLEG): container finished" podID="47ea3616-f54d-4603-a56f-5c549d66eb81" containerID="7ce414a2fe5b47ffe1fb85420f1a1a2ee9b0658559c4aa64acfdaea98ed6cdcf" exitCode=0
Nov 03 00:05:17.302417 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.302396    5012 manager.go:324] Recovery completed
Nov 03 00:05:17.322224 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.322204    5012 cpu_manager.go:226] "Starting" policy="none"
Nov 03 00:05:17.322224 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.322217    5012 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 00:05:17.322339 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.322233    5012 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 00:05:17.322383 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.322352    5012 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 00:05:17.322383 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.322363    5012 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 00:05:17.322383 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.322379    5012 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 00:05:17.322477 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.322384    5012 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 00:05:17.322477 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.322399    5012 policy_none.go:50] "Start"
Nov 03 00:05:17.322477 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.322412    5012 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 00:05:17.322477 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.322421    5012 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 00:05:17.322578 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.322543    5012 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 00:05:17.322578 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.322550    5012 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 00:05:17.322578 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.322555    5012 policy_none.go:44] "Start"
Nov 03 00:05:17.328193 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.328162    5012 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 00:05:17.330862 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.330843    5012 manager.go:341] "Starting Device Plugin manager"
Nov 03 00:05:17.331568 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:17.330877    5012 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 00:05:17.331631 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.331575    5012 server.go:87] "Starting device plugin registration server"
Nov 03 00:05:17.331937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.331923    5012 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 00:05:17.331974 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.331940    5012 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 00:05:17.332034 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.332013    5012 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 00:05:17.332132 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.332105    5012 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 00:05:17.332182 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.332140    5012 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 00:05:17.332182 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.332148    5012 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 00:05:17.332288 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.332183    5012 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T00:05:17.332130101Z","UUID":"aa76e276-4285-453a-8d7d-4a199f6642e6","Handler":null,"Name":"","Endpoint":""}
Nov 03 00:05:17.333582 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.333549    5012 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 00:05:17.333582 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.333576    5012 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 00:05:17.333711 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:17.333686    5012 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 00:05:17.336213 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.335889    5012 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-161.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal"]
Nov 03 00:05:17.336683 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.336667    5012 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-161.ec2.internal"
Nov 03 00:05:17.336746 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.336691    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal" event={"ID":"317fa55400daeb49ee1d8f32b909a6db","Type":"ContainerStarted","Data":"5d03f4f00e77eb15f0a30517b16c8fd85e30f4edb20925739e071fc619db565b"}
Nov 03 00:05:17.336746 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.336733    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal" event={"ID":"317fa55400daeb49ee1d8f32b909a6db","Type":"ContainerDied","Data":"2c5b611bbafcad9f4d61ebaad27fd448fe8c66a6ae02abca20f4216e2f97c366"}
Nov 03 00:05:17.336815 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.336752    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal" event={"ID":"317fa55400daeb49ee1d8f32b909a6db","Type":"ContainerStarted","Data":"25b8cd2ec77476e68d59d598da6f50bc0901483679ddc07512611488c8c5469e"}
Nov 03 00:05:17.336815 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.336794    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-161.ec2.internal" event={"ID":"b4a90a0b8aa58f888d21195c4bcfcc5c","Type":"ContainerStarted","Data":"302825afb86b779681d3c71625afa8883a299c5f3648182bc72a1ed7ca77c2ad"}
Nov 03 00:05:17.336868 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.336812    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-161.ec2.internal" event={"ID":"b4a90a0b8aa58f888d21195c4bcfcc5c","Type":"ContainerStarted","Data":"32925b0fda6815092f415eee6ae94fde3490062f322d962538f8ff5fcf9c953a"}
Nov 03 00:05:17.336999 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.336984    5012 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal"
Nov 03 00:05:17.357783 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.357758    5012 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 00:05:17.357891 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:17.357807    5012 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-161.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-161.ec2.internal"
Nov 03 00:05:17.358603 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.358585    5012 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 00:05:17.358673 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:17.358614    5012 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal"
Nov 03 00:05:17.432714 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.432685    5012 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:05:17.435109 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.435082    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b4a90a0b8aa58f888d21195c4bcfcc5c-config\") pod \"kube-apiserver-proxy-ip-10-0-133-161.ec2.internal\" (UID: \"b4a90a0b8aa58f888d21195c4bcfcc5c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-161.ec2.internal"
Nov 03 00:05:17.435209 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.435111    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/317fa55400daeb49ee1d8f32b909a6db-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal\" (UID: \"317fa55400daeb49ee1d8f32b909a6db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal"
Nov 03 00:05:17.435209 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.435131    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/317fa55400daeb49ee1d8f32b909a6db-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal\" (UID: \"317fa55400daeb49ee1d8f32b909a6db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal"
Nov 03 00:05:17.436238 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.436215    5012 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:05:17.436298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.436255    5012 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:05:17.436298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.436265    5012 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-161.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:05:17.436367 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.436306    5012 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-161.ec2.internal"
Nov 03 00:05:17.468270 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.468216    5012 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-161.ec2.internal"
Nov 03 00:05:17.468350 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.468274    5012 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-161.ec2.internal"
Nov 03 00:05:17.536092 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.536064    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b4a90a0b8aa58f888d21195c4bcfcc5c-config\") pod \"kube-apiserver-proxy-ip-10-0-133-161.ec2.internal\" (UID: \"b4a90a0b8aa58f888d21195c4bcfcc5c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-161.ec2.internal"
Nov 03 00:05:17.536092 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.536079    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b4a90a0b8aa58f888d21195c4bcfcc5c-config\") pod \"kube-apiserver-proxy-ip-10-0-133-161.ec2.internal\" (UID: \"b4a90a0b8aa58f888d21195c4bcfcc5c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-161.ec2.internal"
Nov 03 00:05:17.536289 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.536111    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/317fa55400daeb49ee1d8f32b909a6db-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal\" (UID: \"317fa55400daeb49ee1d8f32b909a6db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal"
Nov 03 00:05:17.536289 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.536135    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/317fa55400daeb49ee1d8f32b909a6db-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal\" (UID: \"317fa55400daeb49ee1d8f32b909a6db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal"
Nov 03 00:05:17.536289 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.536183    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/317fa55400daeb49ee1d8f32b909a6db-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal\" (UID: \"317fa55400daeb49ee1d8f32b909a6db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal"
Nov 03 00:05:17.536289 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:17.536201    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/317fa55400daeb49ee1d8f32b909a6db-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal\" (UID: \"317fa55400daeb49ee1d8f32b909a6db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal"
Nov 03 00:05:18.186633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.186600    5012 apiserver.go:52] "Watching apiserver"
Nov 03 00:05:18.244608 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.244582    5012 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 00:05:18.245253 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.245230    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-rtkhb","openshift-image-registry/image-pruner-29894400-s5xp5","openshift-network-console/networking-console-plugin-86c478bb88-2m6bp","openshift-kube-storage-version-migrator/migrator-66df5d496b-94gh8","openshift-ovn-kubernetes/ovnkube-node-xbrpr","kube-system/kube-apiserver-proxy-ip-10-0-133-161.ec2.internal","openshift-ingress-canary/ingress-canary-n68xd","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc","openshift-image-registry/image-registry-668bbfbdd4-9m8pn","openshift-ingress/router-default-7df847f57-khf2f","openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4","openshift-insights/insights-operator-7c4986d449-kzsm6","openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p","kube-system/global-pull-secret-syncer-bgpx8","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v4sr6","openshift-console-operator/console-operator-7cd56b8dbc-5d95d","openshift-image-registry/image-registry-7564dc7d49-qtdkm","openshift-insights/insights-runtime-extractor-xwdjk","kube-system/konnectivity-agent-ld685","openshift-dns/dns-default-lcdlg","openshift-dns/node-resolver-gtngg","openshift-network-diagnostics/network-check-target-ttc95","openshift-cluster-node-tuning-operator/tuned-7fxg5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-161.ec2.internal","openshift-multus/multus-additional-cni-plugins-bk4mx","openshift-multus/network-metrics-daemon-phwfk","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6","openshift-network-diagnostics/network-check-source-6f78d59f99-8mjkf","openshift-network-operator/iptables-alerter-f4p5q","kube-system/kas-connection-checker-685895c7c9-lnk6s","openshift-multus/multus-bcgzj","kube-system/kas-connection-checker-685895c7c9-2mp9l","kube-system/kas-connection-checker-685895c7c9-lt4f5"]
Nov 03 00:05:18.246821 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.246800    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:18.247270 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.247248    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:18.248265 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.248250    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"
Nov 03 00:05:18.249399 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.249381    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"
Nov 03 00:05:18.250286 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.250272    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2m6bp"
Nov 03 00:05:18.256405 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.256388    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:05:18.258667 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.258644    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.258751 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.258684    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.259794 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.259776    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:18.259965 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.259951    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.260063 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.260045    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.260356 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.260343    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:18.260818 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.260801    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 00:05:18.261007 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.260971    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 03 00:05:18.261079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.261010    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 00:05:18.261079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.261059    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.261210 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.261195    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.261266 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.261248    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s7x9g\""
Nov 03 00:05:18.261266 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.261261    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 03 00:05:18.261368 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.261305    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.261368 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.261325    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 03 00:05:18.261735 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.261722    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:18.270110 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.270093    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 03 00:05:18.270195 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.270110    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 00:05:18.270418 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.270403    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 03 00:05:18.270469 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.270440    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 03 00:05:18.270469 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.270449    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.270614 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.270600    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 03 00:05:18.270894 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.270867    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 00:05:18.270894 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.270885    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.271030 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.270899    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 00:05:18.271030 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.270945    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-rbmkp\""
Nov 03 00:05:18.271340 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.271327    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 00:05:18.271944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.271931    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 00:05:18.272411 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.272396    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 03 00:05:18.280018 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.280002    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 03 00:05:18.280345 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.280319    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 03 00:05:18.280405 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.280344    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 03 00:05:18.280405 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.280370    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 03 00:05:18.280405 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.280390    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 00:05:18.280536 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.280509    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xfsv5\""
Nov 03 00:05:18.280574 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.280529    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 00:05:18.280574 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.280556    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-wvfrg\""
Nov 03 00:05:18.280658 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.280575    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-tbknj\""
Nov 03 00:05:18.280658 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.280590    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-hcj5z\""
Nov 03 00:05:18.280658 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.280603    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-shzgc\""
Nov 03 00:05:18.280658 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.280609    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rns5k\""
Nov 03 00:05:18.280658 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.280629    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pr6ld\""
Nov 03 00:05:18.280811 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.280657    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-vm5wz\""
Nov 03 00:05:18.280905 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.280891    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 00:05:18.280905 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.280897    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8v79z\""
Nov 03 00:05:18.281011 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.280949    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 03 00:05:18.281059 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281012    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wsqcw\""
Nov 03 00:05:18.281059 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281053    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tqxtt\""
Nov 03 00:05:18.281166 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281061    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 00:05:18.281166 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281097    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 00:05:18.281166 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281108    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dlnxq\""
Nov 03 00:05:18.281166 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281136    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-nrnbx\""
Nov 03 00:05:18.281166 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281151    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.281166 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281158    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.281166 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281163    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gt4zj\""
Nov 03 00:05:18.281486 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281185    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.281486 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281187    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jtdbf\""
Nov 03 00:05:18.281486 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281260    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.281486 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281262    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.281486 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281320    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.281486 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281343    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2dzkq\""
Nov 03 00:05:18.281486 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281360    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.281486 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281363    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 03 00:05:18.281486 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281381    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\""
Nov 03 00:05:18.281486 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281421    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 03 00:05:18.281486 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281424    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 00:05:18.281486 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281436    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.281486 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281439    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.281934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281512    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.281934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281489    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.281934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281583    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.281934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281643    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 03 00:05:18.281934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281650    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.281934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281664    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.282495 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.282473    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.282693 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.281585    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 00:05:18.282832 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.282812    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 00:05:18.283074 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.283055    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 00:05:18.283150 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.283075    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 03 00:05:18.283202 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.283136    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 00:05:18.283248 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.283236    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.283323 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.283306    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 00:05:18.283368 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.283061    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.283412 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.283315    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.283554 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.283537    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 00:05:18.283653 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.283638    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.283701 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.283679    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.283814 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.283798    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 00:05:18.283956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.283941    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 03 00:05:18.284114 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.284097    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qr8z8\""
Nov 03 00:05:18.284174 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.284112    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 00:05:18.284247 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.284218    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.284304 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.284285    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.284457 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.284439    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.284567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.284434    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.284672 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.284473    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.284904 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.284889    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 00:05:18.285175 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.285156    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ct6nh\""
Nov 03 00:05:18.285255 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.285188    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7llq5\""
Nov 03 00:05:18.285255 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.285203    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 00:05:18.285255 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.285160    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7b8cw\""
Nov 03 00:05:18.285394 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.285291    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 00:05:18.285439 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.285387    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 00:05:18.285491 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.285444    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4v7ss\""
Nov 03 00:05:18.285491 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.285472    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.285576 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.285537    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.285576 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.285536    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.285576 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.285565    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 00:05:18.285692 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.285675    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.285756 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.285738    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9mnsd\""
Nov 03 00:05:18.291673 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.291659    5012 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-161.ec2.internal"
Nov 03 00:05:18.303876 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.303858    5012 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 00:05:18.308183 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.308167    5012 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 00:05:18.308237 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.308208    5012 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-161.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-161.ec2.internal"
Nov 03 00:05:18.339668 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.339645    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-registration-dir\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:05:18.339761 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.339680    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-hcrh6\" (UID: \"a8dde726-eabf-405e-9d38-a110274353ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"
Nov 03 00:05:18.339761 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.339707    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnl7p\" (UniqueName: \"kubernetes.io/projected/a8dde726-eabf-405e-9d38-a110274353ca-kube-api-access-jnl7p\") pod \"cluster-samples-operator-6ddb6c8b94-hcrh6\" (UID: \"a8dde726-eabf-405e-9d38-a110274353ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"
Nov 03 00:05:18.339861 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.339830    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6aecdeeb-4efb-413e-a26c-9270468062a8-original-pull-secret\") pod \"global-pull-secret-syncer-bgpx8\" (UID: \"6aecdeeb-4efb-413e-a26c-9270468062a8\") " pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:05:18.339931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.339868    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-image-registry-private-configuration\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:18.339931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.339901    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-certificates\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:18.340030 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.339951    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-hcrh6\" (UID: \"a8dde726-eabf-405e-9d38-a110274353ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"
Nov 03 00:05:18.340071 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340039    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-var-lib-kubelet\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.340106 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340068    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-systemd-units\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.340106 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340091    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-cni-bin\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.340106 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340090    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-certificates\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:18.340249 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340123    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-image-registry-private-configuration\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:18.340249 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340177    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47ea3616-f54d-4603-a56f-5c549d66eb81-cnibin\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:05:18.340249 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340209    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-multus-cni-dir\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.340249 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340237    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6aecdeeb-4efb-413e-a26c-9270468062a8-original-pull-secret\") pod \"global-pull-secret-syncer-bgpx8\" (UID: \"6aecdeeb-4efb-413e-a26c-9270468062a8\") " pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:05:18.340358 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340240    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7b5n\" (UniqueName: \"kubernetes.io/projected/4c5d55cb-0306-4dc1-b25b-c2444f148af6-kube-api-access-x7b5n\") pod \"migrator-66df5d496b-94gh8\" (UID: \"4c5d55cb-0306-4dc1-b25b-c2444f148af6\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-94gh8"
Nov 03 00:05:18.340358 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340301    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:18.340358 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340322    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-etc-kubernetes\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.340434 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340362    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-device-dir\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:05:18.340434 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340403    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-sysconfig\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.340494 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340448    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-lib-modules\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.340494 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340468    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-run-k8s-cni-cncf-io\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.340494 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340482    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-run-multus-certs\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.340571 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340506    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6aecdeeb-4efb-413e-a26c-9270468062a8-dbus\") pod \"global-pull-secret-syncer-bgpx8\" (UID: \"6aecdeeb-4efb-413e-a26c-9270468062a8\") " pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:05:18.340571 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340534    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e0cef7-ba0d-4546-a193-2f411c441350-tmp-dir\") pod \"node-resolver-gtngg\" (UID: \"28e0cef7-ba0d-4546-a193-2f411c441350\") " pod="openshift-dns/node-resolver-gtngg"
Nov 03 00:05:18.340571 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340561    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffkwv\" (UID: \"dbea6432-5ba3-49f3-8a83-db2605a3aa2c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"
Nov 03 00:05:18.340659 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340578    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-run-ovn\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.340659 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340623    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-tmp\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:18.340659 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340653    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-etc-selinux\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:05:18.340739 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340654    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e0cef7-ba0d-4546-a193-2f411c441350-tmp-dir\") pod \"node-resolver-gtngg\" (UID: \"28e0cef7-ba0d-4546-a193-2f411c441350\") " pod="openshift-dns/node-resolver-gtngg"
Nov 03 00:05:18.340739 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340670    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-ca-trust-extracted\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:18.340739 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340724    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-ca-trust-extracted\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:18.340739 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340727    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-systemd\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.340842 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340736    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-tmp\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:18.340842 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340764    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/735e8903-5dfe-4cb2-8937-cfc3d09ffa36-cert\") pod \"ingress-canary-n68xd\" (UID: \"735e8903-5dfe-4cb2-8937-cfc3d09ffa36\") " pod="openshift-ingress-canary/ingress-canary-n68xd"
Nov 03 00:05:18.340842 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340781    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-image-registry-private-configuration\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:18.340842 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340834    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:18.340965 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340951    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:18.341001 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340976    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/735e8903-5dfe-4cb2-8937-cfc3d09ffa36-cert\") pod \"ingress-canary-n68xd\" (UID: \"735e8903-5dfe-4cb2-8937-cfc3d09ffa36\") " pod="openshift-ingress-canary/ingress-canary-n68xd"
Nov 03 00:05:18.341001 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340982    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-image-registry-private-configuration\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:18.341001 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.340992    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bnks\" (UniqueName: \"kubernetes.io/projected/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-api-access-6bnks\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:18.341081 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.341020    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-sys\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.341110 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.341083    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kgwx\" (UniqueName: \"kubernetes.io/projected/d8606abd-d458-4bee-b150-461becfbb83b-kube-api-access-4kgwx\") pod \"kas-connection-checker-685895c7c9-lnk6s\" (UID: \"d8606abd-d458-4bee-b150-461becfbb83b\") " pod="kube-system/kas-connection-checker-685895c7c9-lnk6s"
Nov 03 00:05:18.341138 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.341116    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6cb79353-f35d-4dde-ad87-f0d03352cc49-env-overrides\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.341193 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.341175    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-certificates\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:18.341232 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.341221    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-var-lib-kubelet\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.341262 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.341246    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jr76\" (UniqueName: \"kubernetes.io/projected/a173095c-7c94-4533-8529-81b464886f05-kube-api-access-8jr76\") pod \"network-metrics-daemon-phwfk\" (UID: \"a173095c-7c94-4533-8529-81b464886f05\") " pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:05:18.341290 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.341271    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6cb79353-f35d-4dde-ad87-f0d03352cc49-env-overrides\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.341366 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.341302    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/28e0cef7-ba0d-4546-a193-2f411c441350-hosts-file\") pod \"node-resolver-gtngg\" (UID: \"28e0cef7-ba0d-4546-a193-2f411c441350\") " pod="openshift-dns/node-resolver-gtngg"
Nov 03 00:05:18.341409 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.341388    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/689b30e8-936e-49a4-a775-4ce908ad1ab8-config\") pod \"service-ca-operator-544d749c8c-w6k5p\" (UID: \"689b30e8-936e-49a4-a775-4ce908ad1ab8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p"
Nov 03 00:05:18.341409 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.341378    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-certificates\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:18.341505 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.341436    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-run-openvswitch\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.341505 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.341485    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6cb79353-f35d-4dde-ad87-f0d03352cc49-ovnkube-config\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.341588 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.341535    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9pbs\" (UniqueName: \"kubernetes.io/projected/689b30e8-936e-49a4-a775-4ce908ad1ab8-kube-api-access-h9pbs\") pod \"service-ca-operator-544d749c8c-w6k5p\" (UID: \"689b30e8-936e-49a4-a775-4ce908ad1ab8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p"
Nov 03 00:05:18.341588 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.341555    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/689b30e8-936e-49a4-a775-4ce908ad1ab8-config\") pod \"service-ca-operator-544d749c8c-w6k5p\" (UID: \"689b30e8-936e-49a4-a775-4ce908ad1ab8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p"
Nov 03 00:05:18.341677 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.341644    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-node-log\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.341677 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.341658    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6cb79353-f35d-4dde-ad87-f0d03352cc49-ovnkube-config\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.341778 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.341689    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6cb79353-f35d-4dde-ad87-f0d03352cc49-ovn-node-metrics-cert\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.341778 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.341719    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47ea3616-f54d-4603-a56f-5c549d66eb81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:05:18.341778 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.341747    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38b66660-5608-4e82-a1eb-db3d6ce57ca5-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc\" (UID: \"38b66660-5608-4e82-a1eb-db3d6ce57ca5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc"
Nov 03 00:05:18.341778 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.341769    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:18.342000 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.341811    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/daa3da9a-3a5f-49a4-8a09-0036fdc5aa44-serviceca\") pod \"image-pruner-29894400-s5xp5\" (UID: \"daa3da9a-3a5f-49a4-8a09-0036fdc5aa44\") " pod="openshift-image-registry/image-pruner-29894400-s5xp5"
Nov 03 00:05:18.342000 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.341871    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-slash\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.342000 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.341884    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6cb79353-f35d-4dde-ad87-f0d03352cc49-ovn-node-metrics-cert\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.342000 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.341966    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.342000 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.341996    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rwgq\" (UniqueName: \"kubernetes.io/projected/6cb79353-f35d-4dde-ad87-f0d03352cc49-kube-api-access-7rwgq\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.342204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342000    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/daa3da9a-3a5f-49a4-8a09-0036fdc5aa44-serviceca\") pod \"image-pruner-29894400-s5xp5\" (UID: \"daa3da9a-3a5f-49a4-8a09-0036fdc5aa44\") " pod="openshift-image-registry/image-pruner-29894400-s5xp5"
Nov 03 00:05:18.342204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342000    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38b66660-5608-4e82-a1eb-db3d6ce57ca5-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc\" (UID: \"38b66660-5608-4e82-a1eb-db3d6ce57ca5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc"
Nov 03 00:05:18.342204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342038    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-system-cni-dir\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.342204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342063    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/86d13ddf-e3f4-49ad-a779-b9590f50076d-data-volume\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:18.342204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342077    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c819325-a41e-445e-858d-fa52911383fe-tmp\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.342204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342142    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86d13ddf-e3f4-49ad-a779-b9590f50076d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:18.342204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342149    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/86d13ddf-e3f4-49ad-a779-b9590f50076d-data-volume\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:18.342204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342156    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c819325-a41e-445e-858d-fa52911383fe-tmp\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.342204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342177    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47258bdf-947a-4a28-86f4-02871f64941a-agent-certs\") pod \"konnectivity-agent-ld685\" (UID: \"47258bdf-947a-4a28-86f4-02871f64941a\") " pod="kube-system/konnectivity-agent-ld685"
Nov 03 00:05:18.342204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342201    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47258bdf-947a-4a28-86f4-02871f64941a-konnectivity-ca\") pod \"konnectivity-agent-ld685\" (UID: \"47258bdf-947a-4a28-86f4-02871f64941a\") " pod="kube-system/konnectivity-agent-ld685"
Nov 03 00:05:18.342588 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342255    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-sysctl-d\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.342588 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342310    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkvnm\" (UniqueName: \"kubernetes.io/projected/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-kube-api-access-dkvnm\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:18.342588 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342328    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-log-socket\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.342588 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342358    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xcfc\" (UniqueName: \"kubernetes.io/projected/12eb002e-9765-4443-b488-6cec02b9c961-kube-api-access-5xcfc\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:18.342588 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342374    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c1de1047-c4c9-4d0b-8f72-bafc79987c2e-trusted-ca\") pod \"console-operator-7cd56b8dbc-5d95d\" (UID: \"c1de1047-c4c9-4d0b-8f72-bafc79987c2e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:18.342588 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342373    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47258bdf-947a-4a28-86f4-02871f64941a-konnectivity-ca\") pod \"konnectivity-agent-ld685\" (UID: \"47258bdf-947a-4a28-86f4-02871f64941a\") " pod="kube-system/konnectivity-agent-ld685"
Nov 03 00:05:18.342588 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342432    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chvlq\" (UniqueName: \"kubernetes.io/projected/61d5adce-3738-4cad-aace-39d9d33a739a-kube-api-access-chvlq\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:05:18.342588 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342492    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47258bdf-947a-4a28-86f4-02871f64941a-agent-certs\") pod \"konnectivity-agent-ld685\" (UID: \"47258bdf-947a-4a28-86f4-02871f64941a\") " pod="kube-system/konnectivity-agent-ld685"
Nov 03 00:05:18.342588 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342499    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-255z4\" (UniqueName: \"kubernetes.io/projected/47ea3616-f54d-4603-a56f-5c549d66eb81-kube-api-access-255z4\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:05:18.342895 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342589    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-multus-socket-dir-parent\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.342895 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342695    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ffkwv\" (UID: \"dbea6432-5ba3-49f3-8a83-db2605a3aa2c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"
Nov 03 00:05:18.342895 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342721    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2m6bp\" (UID: \"bb5193c7-10f4-4436-9c69-20e6087439c2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2m6bp"
Nov 03 00:05:18.342895 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342738    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6aecdeeb-4efb-413e-a26c-9270468062a8-kubelet-config\") pod \"global-pull-secret-syncer-bgpx8\" (UID: \"6aecdeeb-4efb-413e-a26c-9270468062a8\") " pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:05:18.342895 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342797    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c1de1047-c4c9-4d0b-8f72-bafc79987c2e-trusted-ca\") pod \"console-operator-7cd56b8dbc-5d95d\" (UID: \"c1de1047-c4c9-4d0b-8f72-bafc79987c2e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:18.342895 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342859    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52tkm\" (UniqueName: \"kubernetes.io/projected/7c819325-a41e-445e-858d-fa52911383fe-kube-api-access-52tkm\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.342895 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342887    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs\") pod \"network-metrics-daemon-phwfk\" (UID: \"a173095c-7c94-4533-8529-81b464886f05\") " pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:05:18.343283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342927    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/689b30e8-936e-49a4-a775-4ce908ad1ab8-serving-cert\") pod \"service-ca-operator-544d749c8c-w6k5p\" (UID: \"689b30e8-936e-49a4-a775-4ce908ad1ab8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p"
Nov 03 00:05:18.343283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342976    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-run-systemd\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.343283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.342983    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ffkwv\" (UID: \"dbea6432-5ba3-49f3-8a83-db2605a3aa2c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"
Nov 03 00:05:18.343283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343000    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-bound-sa-token\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:18.343283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343062    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/689b30e8-936e-49a4-a775-4ce908ad1ab8-serving-cert\") pod \"service-ca-operator-544d749c8c-w6k5p\" (UID: \"689b30e8-936e-49a4-a775-4ce908ad1ab8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p"
Nov 03 00:05:18.343283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343103    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1de1047-c4c9-4d0b-8f72-bafc79987c2e-config\") pod \"console-operator-7cd56b8dbc-5d95d\" (UID: \"c1de1047-c4c9-4d0b-8f72-bafc79987c2e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:18.343283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343132    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw7ww\" (UniqueName: \"kubernetes.io/projected/c1de1047-c4c9-4d0b-8f72-bafc79987c2e-kube-api-access-kw7ww\") pod \"console-operator-7cd56b8dbc-5d95d\" (UID: \"c1de1047-c4c9-4d0b-8f72-bafc79987c2e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:18.343283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343149    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7dvp\" (UniqueName: \"kubernetes.io/projected/d683af01-29f3-4634-ae6d-3de8a5c4fd85-kube-api-access-k7dvp\") pod \"network-check-target-ttc95\" (UID: \"d683af01-29f3-4634-ae6d-3de8a5c4fd85\") " pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:05:18.343283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343167    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-socket-dir\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:05:18.343283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343191    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kscjb\" (UniqueName: \"kubernetes.io/projected/daa3da9a-3a5f-49a4-8a09-0036fdc5aa44-kube-api-access-kscjb\") pod \"image-pruner-29894400-s5xp5\" (UID: \"daa3da9a-3a5f-49a4-8a09-0036fdc5aa44\") " pod="openshift-image-registry/image-pruner-29894400-s5xp5"
Nov 03 00:05:18.343283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343218    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47ea3616-f54d-4603-a56f-5c549d66eb81-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:05:18.343283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343242    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-tmp-dir\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:18.343283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343255    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1de1047-c4c9-4d0b-8f72-bafc79987c2e-config\") pod \"console-operator-7cd56b8dbc-5d95d\" (UID: \"c1de1047-c4c9-4d0b-8f72-bafc79987c2e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:18.343283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343266    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-stats-auth\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:18.343283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343290    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-cni-binary-copy\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343303    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-tmp-dir\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343313    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/86d13ddf-e3f4-49ad-a779-b9590f50076d-crio-socket\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343339    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-serving-cert\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343364    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jkxn\" (UniqueName: \"kubernetes.io/projected/ee7974f8-92b4-498a-bf54-2c3a353f4b49-kube-api-access-6jkxn\") pod \"kas-connection-checker-685895c7c9-2mp9l\" (UID: \"ee7974f8-92b4-498a-bf54-2c3a353f4b49\") " pod="kube-system/kas-connection-checker-685895c7c9-2mp9l"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343388    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-modprobe-d\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343443    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-stats-auth\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343445    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chv2w\" (UniqueName: \"kubernetes.io/projected/38b66660-5608-4e82-a1eb-db3d6ce57ca5-kube-api-access-chv2w\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc\" (UID: \"38b66660-5608-4e82-a1eb-db3d6ce57ca5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343478    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmwfj\" (UniqueName: \"kubernetes.io/projected/28e0cef7-ba0d-4546-a193-2f411c441350-kube-api-access-rmwfj\") pod \"node-resolver-gtngg\" (UID: \"28e0cef7-ba0d-4546-a193-2f411c441350\") " pod="openshift-dns/node-resolver-gtngg"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343497    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343512    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-service-ca-bundle\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343528    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hcrh6\" (UID: \"a8dde726-eabf-405e-9d38-a110274353ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343541    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-serving-cert\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343545    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wm66\" (UniqueName: \"kubernetes.io/projected/913d1e5a-3dfc-4282-b5f0-95c4484b31da-kube-api-access-2wm66\") pod \"kas-connection-checker-685895c7c9-lt4f5\" (UID: \"913d1e5a-3dfc-4282-b5f0-95c4484b31da\") " pod="kube-system/kas-connection-checker-685895c7c9-lt4f5"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343579    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47ea3616-f54d-4603-a56f-5c549d66eb81-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343591    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-cni-binary-copy\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343593    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-host\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343651    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6w5f\" (UniqueName: \"kubernetes.io/projected/735e8903-5dfe-4cb2-8937-cfc3d09ffa36-kube-api-access-r6w5f\") pod \"ingress-canary-n68xd\" (UID: \"735e8903-5dfe-4cb2-8937-cfc3d09ffa36\") " pod="openshift-ingress-canary/ingress-canary-n68xd"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343680    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-run-netns\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343711    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-var-lib-cni-multus\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343739    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2tzh\" (UniqueName: \"kubernetes.io/projected/ffb2a03c-414c-4d68-ab86-212c554270e2-kube-api-access-n2tzh\") pod \"node-ca-rtkhb\" (UID: \"ffb2a03c-414c-4d68-ab86-212c554270e2\") " pod="openshift-image-registry/node-ca-rtkhb"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343764    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38b66660-5608-4e82-a1eb-db3d6ce57ca5-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc\" (UID: \"38b66660-5608-4e82-a1eb-db3d6ce57ca5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343779    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-service-ca-bundle\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343795    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f59nd\" (UniqueName: \"kubernetes.io/projected/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-kube-api-access-f59nd\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343819    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343855    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/908118ed-04c5-483d-bb93-61b5d1116259-host-slash\") pod \"iptables-alerter-f4p5q\" (UID: \"908118ed-04c5-483d-bb93-61b5d1116259\") " pod="openshift-network-operator/iptables-alerter-f4p5q"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343874    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c819325-a41e-445e-858d-fa52911383fe-etc-tuned\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343890    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-trusted-ca\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343899    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38b66660-5608-4e82-a1eb-db3d6ce57ca5-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc\" (UID: \"38b66660-5608-4e82-a1eb-db3d6ce57ca5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343945    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-default-certificate\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.343980    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c819325-a41e-445e-858d-fa52911383fe-etc-tuned\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344024    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsh4d\" (UniqueName: \"kubernetes.io/projected/908118ed-04c5-483d-bb93-61b5d1116259-kube-api-access-wsh4d\") pod \"iptables-alerter-f4p5q\" (UID: \"908118ed-04c5-483d-bb93-61b5d1116259\") " pod="openshift-network-operator/iptables-alerter-f4p5q"
Nov 03 00:05:18.344032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344050    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-kubernetes\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344080    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr45n\" (UniqueName: \"kubernetes.io/projected/eddf03ef-53ea-4fc0-9f3f-da7d741cf06a-kube-api-access-wr45n\") pod \"volume-data-source-validator-58c9f5f46d-v4sr6\" (UID: \"eddf03ef-53ea-4fc0-9f3f-da7d741cf06a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v4sr6"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344105    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6cb79353-f35d-4dde-ad87-f0d03352cc49-ovnkube-script-lib\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344078    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344131    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-installation-pull-secrets\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344214    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-cnibin\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344243    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-run-netns\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344257    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-trusted-ca\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344293    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6cb79353-f35d-4dde-ad87-f0d03352cc49-ovnkube-script-lib\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344295    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-default-certificate\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344300    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-hostroot\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344328    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-installation-pull-secrets\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344354    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344380    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-trusted-ca\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344398    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47ea3616-f54d-4603-a56f-5c549d66eb81-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344416    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-multus-daemon-config\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344435    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/908118ed-04c5-483d-bb93-61b5d1116259-iptables-alerter-script\") pod \"iptables-alerter-f4p5q\" (UID: \"908118ed-04c5-483d-bb93-61b5d1116259\") " pod="openshift-network-operator/iptables-alerter-f4p5q"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344462    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-installation-pull-secrets\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344484    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-run\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344509    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffb2a03c-414c-4d68-ab86-212c554270e2-serviceca\") pod \"node-ca-rtkhb\" (UID: \"ffb2a03c-414c-4d68-ab86-212c554270e2\") " pod="openshift-image-registry/node-ca-rtkhb"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344558    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-config-volume\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344609    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47ea3616-f54d-4603-a56f-5c549d66eb81-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344616    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-multus-daemon-config\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344660    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-snapshots\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344686    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-installation-pull-secrets\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344691    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-sysctl-conf\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344725    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-ca-trust-extracted\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344772    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/908118ed-04c5-483d-bb93-61b5d1116259-iptables-alerter-script\") pod \"iptables-alerter-f4p5q\" (UID: \"908118ed-04c5-483d-bb93-61b5d1116259\") " pod="openshift-network-operator/iptables-alerter-f4p5q"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344785    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-config-volume\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344797    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47ea3616-f54d-4603-a56f-5c549d66eb81-system-cni-dir\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344800    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-ca-trust-extracted\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344823    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffb2a03c-414c-4d68-ab86-212c554270e2-serviceca\") pod \"node-ca-rtkhb\" (UID: \"ffb2a03c-414c-4d68-ab86-212c554270e2\") " pod="openshift-image-registry/node-ca-rtkhb"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344833    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffb2a03c-414c-4d68-ab86-212c554270e2-host\") pod \"node-ca-rtkhb\" (UID: \"ffb2a03c-414c-4d68-ab86-212c554270e2\") " pod="openshift-image-registry/node-ca-rtkhb"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344823    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-snapshots\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344862    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47ea3616-f54d-4603-a56f-5c549d66eb81-cni-binary-copy\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344903    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgqk7\" (UniqueName: \"kubernetes.io/projected/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-kube-api-access-rgqk7\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.344979    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-trusted-ca\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.345015    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-sys-fs\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.345047    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47ea3616-f54d-4603-a56f-5c549d66eb81-cni-binary-copy\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.345112    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-bound-sa-token\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.345128    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-run-ovn-kubernetes\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.345155    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-cni-netd\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.345177    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2r5h\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-kube-api-access-s2r5h\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:18.345201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.345213    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-var-lib-cni-bin\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.346330 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.345228    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-etc-openvswitch\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.346330 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.345247    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:18.346330 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.345263    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfdhj\" (UniqueName: \"kubernetes.io/projected/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-kube-api-access-cfdhj\") pod \"cluster-monitoring-operator-884576c57-ffkwv\" (UID: \"dbea6432-5ba3-49f3-8a83-db2605a3aa2c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"
Nov 03 00:05:18.346330 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.345291    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-kubelet\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.346330 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.345307    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-var-lib-openvswitch\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.346330 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.345322    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-multus-conf-dir\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.346330 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.345352    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bb5193c7-10f4-4436-9c69-20e6087439c2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2m6bp\" (UID: \"bb5193c7-10f4-4436-9c69-20e6087439c2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2m6bp"
Nov 03 00:05:18.346330 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.345371    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbzll\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-kube-api-access-bbzll\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:18.346330 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.345387    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9rtk\" (UniqueName: \"kubernetes.io/projected/667223ca-ffbc-4182-beb6-67e68116171f-kube-api-access-t9rtk\") pod \"network-check-source-6f78d59f99-8mjkf\" (UID: \"667223ca-ffbc-4182-beb6-67e68116171f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8mjkf"
Nov 03 00:05:18.346330 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.345410    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1de1047-c4c9-4d0b-8f72-bafc79987c2e-serving-cert\") pod \"console-operator-7cd56b8dbc-5d95d\" (UID: \"c1de1047-c4c9-4d0b-8f72-bafc79987c2e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:18.346330 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.345653    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bb5193c7-10f4-4436-9c69-20e6087439c2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2m6bp\" (UID: \"bb5193c7-10f4-4436-9c69-20e6087439c2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2m6bp"
Nov 03 00:05:18.346330 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.345760    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1de1047-c4c9-4d0b-8f72-bafc79987c2e-serving-cert\") pod \"console-operator-7cd56b8dbc-5d95d\" (UID: \"c1de1047-c4c9-4d0b-8f72-bafc79987c2e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:18.356537 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.356514    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bnks\" (UniqueName: \"kubernetes.io/projected/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-api-access-6bnks\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:18.356999 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.356981    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7b5n\" (UniqueName: \"kubernetes.io/projected/4c5d55cb-0306-4dc1-b25b-c2444f148af6-kube-api-access-x7b5n\") pod \"migrator-66df5d496b-94gh8\" (UID: \"4c5d55cb-0306-4dc1-b25b-c2444f148af6\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-94gh8"
Nov 03 00:05:18.357220 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.357176    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnl7p\" (UniqueName: \"kubernetes.io/projected/a8dde726-eabf-405e-9d38-a110274353ca-kube-api-access-jnl7p\") pod \"cluster-samples-operator-6ddb6c8b94-hcrh6\" (UID: \"a8dde726-eabf-405e-9d38-a110274353ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"
Nov 03 00:05:18.357280 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.357223    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kgwx\" (UniqueName: \"kubernetes.io/projected/d8606abd-d458-4bee-b150-461becfbb83b-kube-api-access-4kgwx\") pod \"kas-connection-checker-685895c7c9-lnk6s\" (UID: \"d8606abd-d458-4bee-b150-461becfbb83b\") " pod="kube-system/kas-connection-checker-685895c7c9-lnk6s"
Nov 03 00:05:18.357338 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.357324    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jr76\" (UniqueName: \"kubernetes.io/projected/a173095c-7c94-4533-8529-81b464886f05-kube-api-access-8jr76\") pod \"network-metrics-daemon-phwfk\" (UID: \"a173095c-7c94-4533-8529-81b464886f05\") " pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:05:18.359229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.359209    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9pbs\" (UniqueName: \"kubernetes.io/projected/689b30e8-936e-49a4-a775-4ce908ad1ab8-kube-api-access-h9pbs\") pod \"service-ca-operator-544d749c8c-w6k5p\" (UID: \"689b30e8-936e-49a4-a775-4ce908ad1ab8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p"
Nov 03 00:05:18.359541 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.359525    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkvnm\" (UniqueName: \"kubernetes.io/projected/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-kube-api-access-dkvnm\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:18.363046 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.363026    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xcfc\" (UniqueName: \"kubernetes.io/projected/12eb002e-9765-4443-b488-6cec02b9c961-kube-api-access-5xcfc\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:18.365087 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.365065    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52tkm\" (UniqueName: \"kubernetes.io/projected/7c819325-a41e-445e-858d-fa52911383fe-kube-api-access-52tkm\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.365202 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.365186    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-255z4\" (UniqueName: \"kubernetes.io/projected/47ea3616-f54d-4603-a56f-5c549d66eb81-kube-api-access-255z4\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:05:18.365322 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.365304    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rwgq\" (UniqueName: \"kubernetes.io/projected/6cb79353-f35d-4dde-ad87-f0d03352cc49-kube-api-access-7rwgq\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.365360 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.365314    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chvlq\" (UniqueName: \"kubernetes.io/projected/61d5adce-3738-4cad-aace-39d9d33a739a-kube-api-access-chvlq\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:05:18.365864 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.365849    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7dvp\" (UniqueName: \"kubernetes.io/projected/d683af01-29f3-4634-ae6d-3de8a5c4fd85-kube-api-access-k7dvp\") pod \"network-check-target-ttc95\" (UID: \"d683af01-29f3-4634-ae6d-3de8a5c4fd85\") " pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:05:18.366890 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.366865    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2r5h\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-kube-api-access-s2r5h\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:18.367021 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.367003    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jkxn\" (UniqueName: \"kubernetes.io/projected/ee7974f8-92b4-498a-bf54-2c3a353f4b49-kube-api-access-6jkxn\") pod \"kas-connection-checker-685895c7c9-2mp9l\" (UID: \"ee7974f8-92b4-498a-bf54-2c3a353f4b49\") " pod="kube-system/kas-connection-checker-685895c7c9-2mp9l"
Nov 03 00:05:18.367344 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.367327    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kscjb\" (UniqueName: \"kubernetes.io/projected/daa3da9a-3a5f-49a4-8a09-0036fdc5aa44-kube-api-access-kscjb\") pod \"image-pruner-29894400-s5xp5\" (UID: \"daa3da9a-3a5f-49a4-8a09-0036fdc5aa44\") " pod="openshift-image-registry/image-pruner-29894400-s5xp5"
Nov 03 00:05:18.367576 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.367561    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chv2w\" (UniqueName: \"kubernetes.io/projected/38b66660-5608-4e82-a1eb-db3d6ce57ca5-kube-api-access-chv2w\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc\" (UID: \"38b66660-5608-4e82-a1eb-db3d6ce57ca5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc"
Nov 03 00:05:18.367856 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.367838    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6w5f\" (UniqueName: \"kubernetes.io/projected/735e8903-5dfe-4cb2-8937-cfc3d09ffa36-kube-api-access-r6w5f\") pod \"ingress-canary-n68xd\" (UID: \"735e8903-5dfe-4cb2-8937-cfc3d09ffa36\") " pod="openshift-ingress-canary/ingress-canary-n68xd"
Nov 03 00:05:18.367888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.367838    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbzll\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-kube-api-access-bbzll\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:18.369143 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.369123    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wm66\" (UniqueName: \"kubernetes.io/projected/913d1e5a-3dfc-4282-b5f0-95c4484b31da-kube-api-access-2wm66\") pod \"kas-connection-checker-685895c7c9-lt4f5\" (UID: \"913d1e5a-3dfc-4282-b5f0-95c4484b31da\") " pod="kube-system/kas-connection-checker-685895c7c9-lt4f5"
Nov 03 00:05:18.369272 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.369247    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgqk7\" (UniqueName: \"kubernetes.io/projected/a0d4ed33-0500-4c65-ba72-b2a584b6bdf2-kube-api-access-rgqk7\") pod \"insights-operator-7c4986d449-kzsm6\" (UID: \"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2\") " pod="openshift-insights/insights-operator-7c4986d449-kzsm6"
Nov 03 00:05:18.369398 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.369275    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-bound-sa-token\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:18.369566 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.369545    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsh4d\" (UniqueName: \"kubernetes.io/projected/908118ed-04c5-483d-bb93-61b5d1116259-kube-api-access-wsh4d\") pod \"iptables-alerter-f4p5q\" (UID: \"908118ed-04c5-483d-bb93-61b5d1116259\") " pod="openshift-network-operator/iptables-alerter-f4p5q"
Nov 03 00:05:18.369628 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.369611    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfdhj\" (UniqueName: \"kubernetes.io/projected/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-kube-api-access-cfdhj\") pod \"cluster-monitoring-operator-884576c57-ffkwv\" (UID: \"dbea6432-5ba3-49f3-8a83-db2605a3aa2c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"
Nov 03 00:05:18.369706 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.369684    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2tzh\" (UniqueName: \"kubernetes.io/projected/ffb2a03c-414c-4d68-ab86-212c554270e2-kube-api-access-n2tzh\") pod \"node-ca-rtkhb\" (UID: \"ffb2a03c-414c-4d68-ab86-212c554270e2\") " pod="openshift-image-registry/node-ca-rtkhb"
Nov 03 00:05:18.369839 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.369825    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw7ww\" (UniqueName: \"kubernetes.io/projected/c1de1047-c4c9-4d0b-8f72-bafc79987c2e-kube-api-access-kw7ww\") pod \"console-operator-7cd56b8dbc-5d95d\" (UID: \"c1de1047-c4c9-4d0b-8f72-bafc79987c2e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:18.370471 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.370453    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmwfj\" (UniqueName: \"kubernetes.io/projected/28e0cef7-ba0d-4546-a193-2f411c441350-kube-api-access-rmwfj\") pod \"node-resolver-gtngg\" (UID: \"28e0cef7-ba0d-4546-a193-2f411c441350\") " pod="openshift-dns/node-resolver-gtngg"
Nov 03 00:05:18.370834 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.370815    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-bound-sa-token\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:18.371211 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.371198    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f59nd\" (UniqueName: \"kubernetes.io/projected/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-kube-api-access-f59nd\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.371332 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.371316    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr45n\" (UniqueName: \"kubernetes.io/projected/eddf03ef-53ea-4fc0-9f3f-da7d741cf06a-kube-api-access-wr45n\") pod \"volume-data-source-validator-58c9f5f46d-v4sr6\" (UID: \"eddf03ef-53ea-4fc0-9f3f-da7d741cf06a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v4sr6"
Nov 03 00:05:18.372534 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.372517    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9rtk\" (UniqueName: \"kubernetes.io/projected/667223ca-ffbc-4182-beb6-67e68116171f-kube-api-access-t9rtk\") pod \"network-check-source-6f78d59f99-8mjkf\" (UID: \"667223ca-ffbc-4182-beb6-67e68116171f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8mjkf"
Nov 03 00:05:18.446373 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446327    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-sysctl-d\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.446373 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446351    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-log-socket\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.446373 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446369    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-multus-socket-dir-parent\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.446567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446383    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2m6bp\" (UID: \"bb5193c7-10f4-4436-9c69-20e6087439c2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2m6bp"
Nov 03 00:05:18.446567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446407    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6aecdeeb-4efb-413e-a26c-9270468062a8-kubelet-config\") pod \"global-pull-secret-syncer-bgpx8\" (UID: \"6aecdeeb-4efb-413e-a26c-9270468062a8\") " pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:05:18.446567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446423    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs\") pod \"network-metrics-daemon-phwfk\" (UID: \"a173095c-7c94-4533-8529-81b464886f05\") " pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:05:18.446567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446436    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-run-systemd\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.446567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446453    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-socket-dir\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:05:18.446567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446468    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-log-socket\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.446567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446479    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/86d13ddf-e3f4-49ad-a779-b9590f50076d-crio-socket\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:18.446567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446467    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-multus-socket-dir-parent\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.446567 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.446498    5012 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 00:05:18.446567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446504    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-modprobe-d\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.446567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446508    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-sysctl-d\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.446567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446492    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6aecdeeb-4efb-413e-a26c-9270468062a8-kubelet-config\") pod \"global-pull-secret-syncer-bgpx8\" (UID: \"6aecdeeb-4efb-413e-a26c-9270468062a8\") " pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:05:18.446567 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.446525    5012 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 00:05:18.446567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446534    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hcrh6\" (UID: \"a8dde726-eabf-405e-9d38-a110274353ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"
Nov 03 00:05:18.446567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446545    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/86d13ddf-e3f4-49ad-a779-b9590f50076d-crio-socket\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:18.446567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446550    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-socket-dir\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:05:18.446567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446559    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-host\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.446567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446503    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-run-systemd\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446586    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-modprobe-d\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.446588    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert podName:bb5193c7-10f4-4436-9c69-20e6087439c2 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:18.946569435 +0000 UTC m=+1.826153466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2m6bp" (UID: "bb5193c7-10f4-4436-9c69-20e6087439c2") : secret "networking-console-plugin-cert" not found
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446615    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-host\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.446621    5012 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.446630    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs podName:a173095c-7c94-4533-8529-81b464886f05 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:18.946623194 +0000 UTC m=+1.826207219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs") pod "network-metrics-daemon-phwfk" (UID: "a173095c-7c94-4533-8529-81b464886f05") : secret "metrics-daemon-secret" not found
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446644    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-run-netns\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.446658    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls podName:a8dde726-eabf-405e-9d38-a110274353ca nodeName:}" failed. No retries permitted until 2026-11-03 00:05:18.946644588 +0000 UTC m=+1.826228613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hcrh6" (UID: "a8dde726-eabf-405e-9d38-a110274353ca") : secret "samples-operator-tls" not found
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446673    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-run-netns\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446685    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-var-lib-cni-multus\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446719    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446741    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/908118ed-04c5-483d-bb93-61b5d1116259-host-slash\") pod \"iptables-alerter-f4p5q\" (UID: \"908118ed-04c5-483d-bb93-61b5d1116259\") " pod="openshift-network-operator/iptables-alerter-f4p5q"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446740    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-var-lib-cni-multus\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.446774    5012 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446781    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/908118ed-04c5-483d-bb93-61b5d1116259-host-slash\") pod \"iptables-alerter-f4p5q\" (UID: \"908118ed-04c5-483d-bb93-61b5d1116259\") " pod="openshift-network-operator/iptables-alerter-f4p5q"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446791    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-kubernetes\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446823    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-cnibin\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.446829    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-rbac-proxy-cm podName:86d13ddf-e3f4-49ad-a779-b9590f50076d nodeName:}" failed. No retries permitted until 2026-11-03 00:05:18.946820328 +0000 UTC m=+1.826404388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xwdjk" (UID: "86d13ddf-e3f4-49ad-a779-b9590f50076d") : configmap "kube-rbac-proxy" not found
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446846    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-run-netns\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446853    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-cnibin\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446869    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-hostroot\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446879    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-kubernetes\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446892    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-run-netns\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446898    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446924    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-hostroot\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446938    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-run\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446958    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-sysctl-conf\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446972    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47ea3616-f54d-4603-a56f-5c549d66eb81-system-cni-dir\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446979    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.446990    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffb2a03c-414c-4d68-ab86-212c554270e2-host\") pod \"node-ca-rtkhb\" (UID: \"ffb2a03c-414c-4d68-ab86-212c554270e2\") " pod="openshift-image-registry/node-ca-rtkhb"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447008    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47ea3616-f54d-4603-a56f-5c549d66eb81-system-cni-dir\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447012    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-sys-fs\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447032    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffb2a03c-414c-4d68-ab86-212c554270e2-host\") pod \"node-ca-rtkhb\" (UID: \"ffb2a03c-414c-4d68-ab86-212c554270e2\") " pod="openshift-image-registry/node-ca-rtkhb"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447033    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-run-ovn-kubernetes\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447011    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-run\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447052    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-cni-netd\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447062    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-run-ovn-kubernetes\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447074    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-var-lib-cni-bin\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447064    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-sys-fs\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447087    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-cni-netd\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447097    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-sysctl-conf\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447103    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-etc-openvswitch\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447127    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447138    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-var-lib-cni-bin\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447151    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-etc-openvswitch\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447167    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-kubelet\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447187    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-var-lib-openvswitch\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447206    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-multus-conf-dir\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447222    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-registration-dir\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447230    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-kubelet\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.447233    5012 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447261    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-var-lib-openvswitch\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.447277    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs podName:12eb002e-9765-4443-b488-6cec02b9c961 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:18.947266548 +0000 UTC m=+1.826850568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs") pod "router-default-7df847f57-khf2f" (UID: "12eb002e-9765-4443-b488-6cec02b9c961") : secret "router-metrics-certs-default" not found
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447277    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-registration-dir\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447271    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-multus-conf-dir\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447296    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-var-lib-kubelet\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447316    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-systemd-units\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447330    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-cni-bin\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.447311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447344    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47ea3616-f54d-4603-a56f-5c549d66eb81-cnibin\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447346    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-systemd-units\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447330    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-var-lib-kubelet\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447358    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-multus-cni-dir\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447380    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447379    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-cni-bin\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447413    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47ea3616-f54d-4603-a56f-5c549d66eb81-cnibin\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447424    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-multus-cni-dir\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.447434    5012 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.447443    5012 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-668bbfbdd4-9m8pn: secret "image-registry-tls" not found
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447449    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-etc-kubernetes\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447470    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-device-dir\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.447482    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls podName:4d7b91ff-203c-4e03-9ad7-9b60215b61b6 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:18.947473877 +0000 UTC m=+1.827057932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls") pod "image-registry-668bbfbdd4-9m8pn" (UID: "4d7b91ff-203c-4e03-9ad7-9b60215b61b6") : secret "image-registry-tls" not found
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447497    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-sysconfig\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447516    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-lib-modules\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447499    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-device-dir\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447531    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-run-k8s-cni-cncf-io\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447535    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-sysconfig\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447528    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-etc-kubernetes\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447564    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-run-multus-certs\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447569    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-run-k8s-cni-cncf-io\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447583    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6aecdeeb-4efb-413e-a26c-9270468062a8-dbus\") pod \"global-pull-secret-syncer-bgpx8\" (UID: \"6aecdeeb-4efb-413e-a26c-9270468062a8\") " pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447602    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffkwv\" (UID: \"dbea6432-5ba3-49f3-8a83-db2605a3aa2c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447620    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-lib-modules\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447630    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-run-multus-certs\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447625    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-run-ovn\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447653    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6aecdeeb-4efb-413e-a26c-9270468062a8-dbus\") pod \"global-pull-secret-syncer-bgpx8\" (UID: \"6aecdeeb-4efb-413e-a26c-9270468062a8\") " pod="kube-system/global-pull-secret-syncer-bgpx8"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447664    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-etc-selinux\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447656    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-run-ovn\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447693    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-systemd\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447706    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/61d5adce-3738-4cad-aace-39d9d33a739a-etc-selinux\") pod \"aws-ebs-csi-driver-node-xx2v4\" (UID: \"61d5adce-3738-4cad-aace-39d9d33a739a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx2v4"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.447671    5012 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447735    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-etc-systemd\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.447765    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls podName:dbea6432-5ba3-49f3-8a83-db2605a3aa2c nodeName:}" failed. No retries permitted until 2026-11-03 00:05:18.947757114 +0000 UTC m=+1.827341139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ffkwv" (UID: "dbea6432-5ba3-49f3-8a83-db2605a3aa2c") : secret "cluster-monitoring-operator-tls" not found
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447722    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447788    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.447791    5012 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.447804    5012 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7564dc7d49-qtdkm: secret "image-registry-tls" not found
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447806    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-sys\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447831    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-var-lib-kubelet\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.447852    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls podName:ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:18.94783785 +0000 UTC m=+1.827421886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls") pod "image-registry-7564dc7d49-qtdkm" (UID: "ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3") : secret "image-registry-tls" not found
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447855    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c819325-a41e-445e-858d-fa52911383fe-sys\") pod \"tuned-7fxg5\" (UID: \"7c819325-a41e-445e-858d-fa52911383fe\") " pod="openshift-cluster-node-tuning-operator/tuned-7fxg5"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447863    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-host-var-lib-kubelet\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.447873    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle podName:12eb002e-9765-4443-b488-6cec02b9c961 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:18.947863103 +0000 UTC m=+1.827447132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle") pod "router-default-7df847f57-khf2f" (UID: "12eb002e-9765-4443-b488-6cec02b9c961") : configmap references non-existent config key: service-ca.crt
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447902    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/28e0cef7-ba0d-4546-a193-2f411c441350-hosts-file\") pod \"node-resolver-gtngg\" (UID: \"28e0cef7-ba0d-4546-a193-2f411c441350\") " pod="openshift-dns/node-resolver-gtngg"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447946    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-run-openvswitch\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447957    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-run-openvswitch\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447970    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-node-log\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447994    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/28e0cef7-ba0d-4546-a193-2f411c441350-hosts-file\") pod \"node-resolver-gtngg\" (UID: \"28e0cef7-ba0d-4546-a193-2f411c441350\") " pod="openshift-dns/node-resolver-gtngg"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447999    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-node-log\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.447996    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47ea3616-f54d-4603-a56f-5c549d66eb81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.448021    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.448036    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-slash\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.448043    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47ea3616-f54d-4603-a56f-5c549d66eb81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bk4mx\" (UID: \"47ea3616-f54d-4603-a56f-5c549d66eb81\") " pod="openshift-multus/multus-additional-cni-plugins-bk4mx"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.448055    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.448079    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-system-cni-dir\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.448084    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-slash\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.448104    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86d13ddf-e3f4-49ad-a779-b9590f50076d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.448120    5012 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.448124    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cb79353-f35d-4dde-ad87-f0d03352cc49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xbrpr\" (UID: \"6cb79353-f35d-4dde-ad87-f0d03352cc49\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.448159    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls podName:7cdb4c55-92a6-4791-8ab8-fcb22ef3a145 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:18.948142883 +0000 UTC m=+1.827726913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls") pod "dns-default-lcdlg" (UID: "7cdb4c55-92a6-4791-8ab8-fcb22ef3a145") : secret "dns-default-metrics-tls" not found
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.448157    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/015c1f9d-d4c4-4749-978a-3cd2166c7bf8-system-cni-dir\") pod \"multus-bcgzj\" (UID: \"015c1f9d-d4c4-4749-978a-3cd2166c7bf8\") " pod="openshift-multus/multus-bcgzj"
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.448188    5012 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:18.449226 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.448213    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86d13ddf-e3f4-49ad-a779-b9590f50076d-insights-runtime-extractor-tls podName:86d13ddf-e3f4-49ad-a779-b9590f50076d nodeName:}" failed. No retries permitted until 2026-11-03 00:05:18.94820534 +0000 UTC m=+1.827789374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86d13ddf-e3f4-49ad-a779-b9590f50076d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xwdjk" (UID: "86d13ddf-e3f4-49ad-a779-b9590f50076d") : secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:18.546977 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.546951    5012 scope.go:122] "RemoveContainer" containerID="82c6612b250a969062f81eafeb8c6730b0ebdea3d3d3cc15571a64b5c5a49b0b"
Nov 03 00:05:18.951805 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.951777    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86d13ddf-e3f4-49ad-a779-b9590f50076d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:18.951932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.951815    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2m6bp\" (UID: \"bb5193c7-10f4-4436-9c69-20e6087439c2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2m6bp"
Nov 03 00:05:18.951932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.951833    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs\") pod \"network-metrics-daemon-phwfk\" (UID: \"a173095c-7c94-4533-8529-81b464886f05\") " pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:05:18.951932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.951857    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hcrh6\" (UID: \"a8dde726-eabf-405e-9d38-a110274353ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"
Nov 03 00:05:18.951932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.951883    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:18.952101 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.951948    5012 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 00:05:18.952101 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.951948    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:18.952101 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.951979    5012 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 00:05:18.952101 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.952006    5012 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 00:05:18.952101 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.952007    5012 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 00:05:18.952101 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.952014    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:18.952101 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.951954    5012 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:18.952101 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.952056    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs podName:12eb002e-9765-4443-b488-6cec02b9c961 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:19.952040246 +0000 UTC m=+2.831624268 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs") pod "router-default-7df847f57-khf2f" (UID: "12eb002e-9765-4443-b488-6cec02b9c961") : secret "router-metrics-certs-default" not found
Nov 03 00:05:18.952101 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.951954    5012 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 00:05:18.952101 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.952097    5012 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 00:05:18.952101 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.952106    5012 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-668bbfbdd4-9m8pn: secret "image-registry-tls" not found
Nov 03 00:05:18.952600 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.952074    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls podName:a8dde726-eabf-405e-9d38-a110274353ca nodeName:}" failed. No retries permitted until 2026-11-03 00:05:19.952064793 +0000 UTC m=+2.831648868 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hcrh6" (UID: "a8dde726-eabf-405e-9d38-a110274353ca") : secret "samples-operator-tls" not found
Nov 03 00:05:18.952600 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.952184    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffkwv\" (UID: \"dbea6432-5ba3-49f3-8a83-db2605a3aa2c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"
Nov 03 00:05:18.952600 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.952218    5012 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 00:05:18.952600 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.952234    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:18.952600 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.952275    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert podName:bb5193c7-10f4-4436-9c69-20e6087439c2 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:19.952251059 +0000 UTC m=+2.831835090 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2m6bp" (UID: "bb5193c7-10f4-4436-9c69-20e6087439c2") : secret "networking-console-plugin-cert" not found
Nov 03 00:05:18.952600 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.952293    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-rbac-proxy-cm podName:86d13ddf-e3f4-49ad-a779-b9590f50076d nodeName:}" failed. No retries permitted until 2026-11-03 00:05:19.952285706 +0000 UTC m=+2.831869726 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xwdjk" (UID: "86d13ddf-e3f4-49ad-a779-b9590f50076d") : configmap "kube-rbac-proxy" not found
Nov 03 00:05:18.952600 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.952302    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86d13ddf-e3f4-49ad-a779-b9590f50076d-insights-runtime-extractor-tls podName:86d13ddf-e3f4-49ad-a779-b9590f50076d nodeName:}" failed. No retries permitted until 2026-11-03 00:05:19.952297133 +0000 UTC m=+2.831881154 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86d13ddf-e3f4-49ad-a779-b9590f50076d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xwdjk" (UID: "86d13ddf-e3f4-49ad-a779-b9590f50076d") : secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:18.952600 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.952313    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls podName:4d7b91ff-203c-4e03-9ad7-9b60215b61b6 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:19.952307658 +0000 UTC m=+2.831891678 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls") pod "image-registry-668bbfbdd4-9m8pn" (UID: "4d7b91ff-203c-4e03-9ad7-9b60215b61b6") : secret "image-registry-tls" not found
Nov 03 00:05:18.952600 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.952329    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs podName:a173095c-7c94-4533-8529-81b464886f05 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:19.952323812 +0000 UTC m=+2.831907833 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs") pod "network-metrics-daemon-phwfk" (UID: "a173095c-7c94-4533-8529-81b464886f05") : secret "metrics-daemon-secret" not found
Nov 03 00:05:18.952600 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.952328    5012 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 00:05:18.952600 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.952343    5012 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7564dc7d49-qtdkm: secret "image-registry-tls" not found
Nov 03 00:05:18.952600 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.952349    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls podName:dbea6432-5ba3-49f3-8a83-db2605a3aa2c nodeName:}" failed. No retries permitted until 2026-11-03 00:05:19.952337298 +0000 UTC m=+2.831921318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ffkwv" (UID: "dbea6432-5ba3-49f3-8a83-db2605a3aa2c") : secret "cluster-monitoring-operator-tls" not found
Nov 03 00:05:18.952600 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.952371    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:18.952600 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.952383    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls podName:ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:19.952366437 +0000 UTC m=+2.831950475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls") pod "image-registry-7564dc7d49-qtdkm" (UID: "ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3") : secret "image-registry-tls" not found
Nov 03 00:05:18.952600 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:18.952427    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:18.952600 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.952435    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle podName:12eb002e-9765-4443-b488-6cec02b9c961 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:19.952429863 +0000 UTC m=+2.832013884 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle") pod "router-default-7df847f57-khf2f" (UID: "12eb002e-9765-4443-b488-6cec02b9c961") : configmap references non-existent config key: service-ca.crt
Nov 03 00:05:18.952600 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.952490    5012 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 00:05:18.952600 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:18.952534    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls podName:7cdb4c55-92a6-4791-8ab8-fcb22ef3a145 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:19.952522474 +0000 UTC m=+2.832106499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls") pod "dns-default-lcdlg" (UID: "7cdb4c55-92a6-4791-8ab8-fcb22ef3a145") : secret "dns-default-metrics-tls" not found
Nov 03 00:05:19.296577 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:19.296524    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 00:05:19.296940 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:19.296886    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/1.log"
Nov 03 00:05:19.296940 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:19.296930    5012 generic.go:356] "Generic (PLEG): container finished" podID="c1de1047-c4c9-4d0b-8f72-bafc79987c2e" containerID="a3f3e8a79c468bc7697b87509d573d8d367915fe57bb7f44c96c4ec1b50d380c" exitCode=255
Nov 03 00:05:19.297015 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:19.296991    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d" event={"ID":"c1de1047-c4c9-4d0b-8f72-bafc79987c2e","Type":"ContainerDied","Data":"a3f3e8a79c468bc7697b87509d573d8d367915fe57bb7f44c96c4ec1b50d380c"}
Nov 03 00:05:19.297050 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:19.297041    5012 scope.go:122] "RemoveContainer" containerID="82c6612b250a969062f81eafeb8c6730b0ebdea3d3d3cc15571a64b5c5a49b0b"
Nov 03 00:05:19.297375 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:19.297356    5012 scope.go:122] "RemoveContainer" containerID="a3f3e8a79c468bc7697b87509d573d8d367915fe57bb7f44c96c4ec1b50d380c"
Nov 03 00:05:19.297522 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:19.297505    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5d95d_openshift-console-operator(c1de1047-c4c9-4d0b-8f72-bafc79987c2e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d" podUID="c1de1047-c4c9-4d0b-8f72-bafc79987c2e"
Nov 03 00:05:19.796119 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:19.796091    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:05:19.960497 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:19.960464    5012 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 00:05:19.960825 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:19.960790    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls podName:7cdb4c55-92a6-4791-8ab8-fcb22ef3a145 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:21.960763391 +0000 UTC m=+4.840347433 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls") pod "dns-default-lcdlg" (UID: "7cdb4c55-92a6-4791-8ab8-fcb22ef3a145") : secret "dns-default-metrics-tls" not found
Nov 03 00:05:19.960825 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:19.960800    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:19.961001 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:19.960902    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86d13ddf-e3f4-49ad-a779-b9590f50076d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:19.961001 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:19.960980    5012 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:19.961111 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:19.961012    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2m6bp\" (UID: \"bb5193c7-10f4-4436-9c69-20e6087439c2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2m6bp"
Nov 03 00:05:19.961111 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:19.961034    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86d13ddf-e3f4-49ad-a779-b9590f50076d-insights-runtime-extractor-tls podName:86d13ddf-e3f4-49ad-a779-b9590f50076d nodeName:}" failed. No retries permitted until 2026-11-03 00:05:21.961019756 +0000 UTC m=+4.840603794 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86d13ddf-e3f4-49ad-a779-b9590f50076d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xwdjk" (UID: "86d13ddf-e3f4-49ad-a779-b9590f50076d") : secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:19.961111 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:19.961076    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs\") pod \"network-metrics-daemon-phwfk\" (UID: \"a173095c-7c94-4533-8529-81b464886f05\") " pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:05:19.961253 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:19.961127    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hcrh6\" (UID: \"a8dde726-eabf-405e-9d38-a110274353ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"
Nov 03 00:05:19.961253 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:19.961143    5012 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 00:05:19.961253 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:19.961165    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:19.961253 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:19.961238    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:19.961438 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:19.961287    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:19.961438 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:19.961305    5012 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 00:05:19.961438 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:19.961308    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert podName:bb5193c7-10f4-4436-9c69-20e6087439c2 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:21.961289818 +0000 UTC m=+4.840873839 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2m6bp" (UID: "bb5193c7-10f4-4436-9c69-20e6087439c2") : secret "networking-console-plugin-cert" not found
Nov 03 00:05:19.961438 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:19.961395    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs podName:a173095c-7c94-4533-8529-81b464886f05 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:21.961379104 +0000 UTC m=+4.840963129 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs") pod "network-metrics-daemon-phwfk" (UID: "a173095c-7c94-4533-8529-81b464886f05") : secret "metrics-daemon-secret" not found
Nov 03 00:05:19.961641 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:19.961443    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffkwv\" (UID: \"dbea6432-5ba3-49f3-8a83-db2605a3aa2c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"
Nov 03 00:05:19.961641 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:19.961490    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:19.961641 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:19.961528    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:19.961782 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:19.961687    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle podName:12eb002e-9765-4443-b488-6cec02b9c961 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:21.961671855 +0000 UTC m=+4.841255876 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle") pod "router-default-7df847f57-khf2f" (UID: "12eb002e-9765-4443-b488-6cec02b9c961") : configmap references non-existent config key: service-ca.crt
Nov 03 00:05:19.961782 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:19.961771    5012 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 00:05:19.961890 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:19.961820    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls podName:dbea6432-5ba3-49f3-8a83-db2605a3aa2c nodeName:}" failed. No retries permitted until 2026-11-03 00:05:21.961807137 +0000 UTC m=+4.841391172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ffkwv" (UID: "dbea6432-5ba3-49f3-8a83-db2605a3aa2c") : secret "cluster-monitoring-operator-tls" not found
Nov 03 00:05:19.961962 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:19.961899    5012 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 00:05:19.962010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:19.961895    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86d13ddf-e3f4-49ad-a779-b9590f50076d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:19.962071 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:19.962018    5012 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 00:05:19.962071 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:19.962024    5012 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7564dc7d49-qtdkm: secret "image-registry-tls" not found
Nov 03 00:05:19.962162 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:19.962072    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls podName:a8dde726-eabf-405e-9d38-a110274353ca nodeName:}" failed. No retries permitted until 2026-11-03 00:05:21.962051958 +0000 UTC m=+4.841635993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hcrh6" (UID: "a8dde726-eabf-405e-9d38-a110274353ca") : secret "samples-operator-tls" not found
Nov 03 00:05:19.962162 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:19.962090    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls podName:ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:21.962080538 +0000 UTC m=+4.841664576 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls") pod "image-registry-7564dc7d49-qtdkm" (UID: "ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3") : secret "image-registry-tls" not found
Nov 03 00:05:19.962255 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:19.962179    5012 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 00:05:19.962255 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:19.962193    5012 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 00:05:19.962255 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:19.962204    5012 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-668bbfbdd4-9m8pn: secret "image-registry-tls" not found
Nov 03 00:05:19.962255 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:19.962231    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs podName:12eb002e-9765-4443-b488-6cec02b9c961 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:21.962208864 +0000 UTC m=+4.841792899 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs") pod "router-default-7df847f57-khf2f" (UID: "12eb002e-9765-4443-b488-6cec02b9c961") : secret "router-metrics-certs-default" not found
Nov 03 00:05:19.962255 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:19.962250    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls podName:4d7b91ff-203c-4e03-9ad7-9b60215b61b6 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:21.962239603 +0000 UTC m=+4.841823629 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls") pod "image-registry-668bbfbdd4-9m8pn" (UID: "4d7b91ff-203c-4e03-9ad7-9b60215b61b6") : secret "image-registry-tls" not found
Nov 03 00:05:20.301490 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:20.301461    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 00:05:21.350281 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:21.350252    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gtngg_28e0cef7-ba0d-4546-a193-2f411c441350/dns-node-resolver/0.log"
Nov 03 00:05:21.550272 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:21.550244    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29894400-s5xp5_daa3da9a-3a5f-49a4-8a09-0036fdc5aa44/image-pruner/0.log"
Nov 03 00:05:21.981648 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:21.981609    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:21.981813 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:21.981654    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffkwv\" (UID: \"dbea6432-5ba3-49f3-8a83-db2605a3aa2c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"
Nov 03 00:05:21.981813 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:21.981699    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:21.981813 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:21.981719    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:21.981813 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:21.981750    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:21.981813 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:21.981780    5012 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 00:05:21.981813 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:21.981802    5012 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-668bbfbdd4-9m8pn: secret "image-registry-tls" not found
Nov 03 00:05:21.982116 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:21.981831    5012 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 00:05:21.982116 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:21.981859    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls podName:4d7b91ff-203c-4e03-9ad7-9b60215b61b6 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:25.981839453 +0000 UTC m=+8.861423488 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls") pod "image-registry-668bbfbdd4-9m8pn" (UID: "4d7b91ff-203c-4e03-9ad7-9b60215b61b6") : secret "image-registry-tls" not found
Nov 03 00:05:21.982116 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:21.981876    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle podName:12eb002e-9765-4443-b488-6cec02b9c961 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:25.98186962 +0000 UTC m=+8.861453641 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle") pod "router-default-7df847f57-khf2f" (UID: "12eb002e-9765-4443-b488-6cec02b9c961") : configmap references non-existent config key: service-ca.crt
Nov 03 00:05:21.982116 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:21.981894    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86d13ddf-e3f4-49ad-a779-b9590f50076d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:21.982116 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:21.981831    5012 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 00:05:21.982116 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:21.981939    5012 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7564dc7d49-qtdkm: secret "image-registry-tls" not found
Nov 03 00:05:21.982116 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:21.981960    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2m6bp\" (UID: \"bb5193c7-10f4-4436-9c69-20e6087439c2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2m6bp"
Nov 03 00:05:21.982116 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:21.981976    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls podName:ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:25.98196323 +0000 UTC m=+8.861547267 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls") pod "image-registry-7564dc7d49-qtdkm" (UID: "ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3") : secret "image-registry-tls" not found
Nov 03 00:05:21.982116 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:21.982004    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls podName:7cdb4c55-92a6-4791-8ab8-fcb22ef3a145 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:25.981993461 +0000 UTC m=+8.861577490 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls") pod "dns-default-lcdlg" (UID: "7cdb4c55-92a6-4791-8ab8-fcb22ef3a145") : secret "dns-default-metrics-tls" not found
Nov 03 00:05:21.982116 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:21.982022    5012 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 00:05:21.982116 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:21.982057    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert podName:bb5193c7-10f4-4436-9c69-20e6087439c2 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:25.982047663 +0000 UTC m=+8.861631703 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2m6bp" (UID: "bb5193c7-10f4-4436-9c69-20e6087439c2") : secret "networking-console-plugin-cert" not found
Nov 03 00:05:21.982116 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:21.982021    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs\") pod \"network-metrics-daemon-phwfk\" (UID: \"a173095c-7c94-4533-8529-81b464886f05\") " pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:05:21.982116 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:21.982074    5012 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 00:05:21.982116 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:21.982100    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hcrh6\" (UID: \"a8dde726-eabf-405e-9d38-a110274353ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"
Nov 03 00:05:21.982116 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:21.982109    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs podName:a173095c-7c94-4533-8529-81b464886f05 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:25.982099741 +0000 UTC m=+8.861683765 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs") pod "network-metrics-daemon-phwfk" (UID: "a173095c-7c94-4533-8529-81b464886f05") : secret "metrics-daemon-secret" not found
Nov 03 00:05:21.982686 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:21.982157    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:21.982686 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:21.982254    5012 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 00:05:21.984393 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:21.984355    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86d13ddf-e3f4-49ad-a779-b9590f50076d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xwdjk\" (UID: \"86d13ddf-e3f4-49ad-a779-b9590f50076d\") " pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:21.984592 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:21.984572    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12eb002e-9765-4443-b488-6cec02b9c961-metrics-certs\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:21.984655 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:21.984601    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbea6432-5ba3-49f3-8a83-db2605a3aa2c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffkwv\" (UID: \"dbea6432-5ba3-49f3-8a83-db2605a3aa2c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"
Nov 03 00:05:21.984655 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:21.984632    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a8dde726-eabf-405e-9d38-a110274353ca-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hcrh6\" (UID: \"a8dde726-eabf-405e-9d38-a110274353ca\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"
Nov 03 00:05:22.080829 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:22.080792    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:22.149099 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:22.149064    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"
Nov 03 00:05:22.150133 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:22.150112    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"
Nov 03 00:05:22.162479 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:22.162455    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xwdjk"
Nov 03 00:05:22.349789 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:22.349761    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rtkhb_ffb2a03c-414c-4d68-ab86-212c554270e2/node-ca/0.log"
Nov 03 00:05:22.458997 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:22.458965    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv"]
Nov 03 00:05:22.462671 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:22.462642    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbea6432_5ba3_49f3_8a83_db2605a3aa2c.slice/crio-5f4fde8e46d4e643be89a747b4ed157e1ea98c276536029e207778a3fa78ba6b WatchSource:0}: Error finding container 5f4fde8e46d4e643be89a747b4ed157e1ea98c276536029e207778a3fa78ba6b: Status 404 returned error can't find the container with id 5f4fde8e46d4e643be89a747b4ed157e1ea98c276536029e207778a3fa78ba6b
Nov 03 00:05:22.464592 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:22.464577    5012 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:05:22.467330 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:22.467310    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xwdjk"]
Nov 03 00:05:22.469502 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:22.469481    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d13ddf_e3f4_49ad_a779_b9590f50076d.slice/crio-998aa89bc47f9c1cd100b150101cbf81de18e1c3ea118ed53d68a06a11f7a121 WatchSource:0}: Error finding container 998aa89bc47f9c1cd100b150101cbf81de18e1c3ea118ed53d68a06a11f7a121: Status 404 returned error can't find the container with id 998aa89bc47f9c1cd100b150101cbf81de18e1c3ea118ed53d68a06a11f7a121
Nov 03 00:05:22.472741 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:22.472123    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6"]
Nov 03 00:05:22.949526 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:22.949490    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n68xd_735e8903-5dfe-4cb2-8937-cfc3d09ffa36/serve-healthcheck-canary/0.log"
Nov 03 00:05:23.315810 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:23.315557    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv" event={"ID":"dbea6432-5ba3-49f3-8a83-db2605a3aa2c","Type":"ContainerStarted","Data":"5f4fde8e46d4e643be89a747b4ed157e1ea98c276536029e207778a3fa78ba6b"}
Nov 03 00:05:23.317142 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:23.317099    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6" event={"ID":"a8dde726-eabf-405e-9d38-a110274353ca","Type":"ContainerStarted","Data":"83f9ae709fde84dca789335722cae1d70b66e796d06d2629be6f66b5de7e3033"}
Nov 03 00:05:23.320307 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:23.320268    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xwdjk" event={"ID":"86d13ddf-e3f4-49ad-a779-b9590f50076d","Type":"ContainerStarted","Data":"24e2001ef95f8c7b81660cbb50976a69e908510eafb883adf20413e78a6be90c"}
Nov 03 00:05:23.320486 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:23.320427    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xwdjk" event={"ID":"86d13ddf-e3f4-49ad-a779-b9590f50076d","Type":"ContainerStarted","Data":"f4564e1016d04506f8bafc9eebfc776fd334918a4ee06aac57a98a8656be3043"}
Nov 03 00:05:23.320486 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:23.320456    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xwdjk" event={"ID":"86d13ddf-e3f4-49ad-a779-b9590f50076d","Type":"ContainerStarted","Data":"998aa89bc47f9c1cd100b150101cbf81de18e1c3ea118ed53d68a06a11f7a121"}
Nov 03 00:05:23.350845 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:23.350800    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-94gh8_4c5d55cb-0306-4dc1-b25b-c2444f148af6/migrator/0.log"
Nov 03 00:05:23.752710 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:23.752673    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc_38b66660-5608-4e82-a1eb-db3d6ce57ca5/kube-storage-version-migrator-operator/0.log"
Nov 03 00:05:23.939762 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:23.939717    5012 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:23.940278 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:23.940257    5012 scope.go:122] "RemoveContainer" containerID="a3f3e8a79c468bc7697b87509d573d8d367915fe57bb7f44c96c4ec1b50d380c"
Nov 03 00:05:23.940526 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:23.940501    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5d95d_openshift-console-operator(c1de1047-c4c9-4d0b-8f72-bafc79987c2e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d" podUID="c1de1047-c4c9-4d0b-8f72-bafc79987c2e"
Nov 03 00:05:24.441363 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:24.441303    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:24.441722 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:24.441704    5012 scope.go:122] "RemoveContainer" containerID="a3f3e8a79c468bc7697b87509d573d8d367915fe57bb7f44c96c4ec1b50d380c"
Nov 03 00:05:24.441962 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:24.441901    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5d95d_openshift-console-operator(c1de1047-c4c9-4d0b-8f72-bafc79987c2e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d" podUID="c1de1047-c4c9-4d0b-8f72-bafc79987c2e"
Nov 03 00:05:25.335271 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:25.335236    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xwdjk" event={"ID":"86d13ddf-e3f4-49ad-a779-b9590f50076d","Type":"ContainerStarted","Data":"d65026753077e07656b8e8f7e8bdb8c111584df69dacacbe29d33bf13c2e8141"}
Nov 03 00:05:25.336775 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:25.336751    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffkwv" event={"ID":"dbea6432-5ba3-49f3-8a83-db2605a3aa2c","Type":"ContainerStarted","Data":"0fffc12c7bc82ab794672a316b372db6934a81dbfe68b6d58aed31b92fe508ae"}
Nov 03 00:05:25.338571 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:25.338548    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6" event={"ID":"a8dde726-eabf-405e-9d38-a110274353ca","Type":"ContainerStarted","Data":"61c2af1fde6cc4b674a9ec8e2c6dadaf889243b41326cc8d1185c0647b44f215"}
Nov 03 00:05:25.338643 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:25.338599    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hcrh6" event={"ID":"a8dde726-eabf-405e-9d38-a110274353ca","Type":"ContainerStarted","Data":"6b7385ff5fc4d0707558edda9eca1218e6cceb3ef39e3ec9fc18efa236e23fad"}
Nov 03 00:05:26.022849 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:26.022808    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:26.022849 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:26.022861    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:26.023148 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:26.022881    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:26.023148 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:26.022902    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:26.023148 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:26.022960    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2m6bp\" (UID: \"bb5193c7-10f4-4436-9c69-20e6087439c2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2m6bp"
Nov 03 00:05:26.023148 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:26.022979    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs\") pod \"network-metrics-daemon-phwfk\" (UID: \"a173095c-7c94-4533-8529-81b464886f05\") " pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:05:26.023148 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:26.023052    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle podName:12eb002e-9765-4443-b488-6cec02b9c961 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:34.023026847 +0000 UTC m=+16.902610891 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle") pod "router-default-7df847f57-khf2f" (UID: "12eb002e-9765-4443-b488-6cec02b9c961") : configmap references non-existent config key: service-ca.crt
Nov 03 00:05:26.023148 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:26.023106    5012 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 00:05:26.023451 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:26.023163    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert podName:bb5193c7-10f4-4436-9c69-20e6087439c2 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:34.023149 +0000 UTC m=+16.902733021 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2m6bp" (UID: "bb5193c7-10f4-4436-9c69-20e6087439c2") : secret "networking-console-plugin-cert" not found
Nov 03 00:05:26.025503 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:26.025476    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdb4c55-92a6-4791-8ab8-fcb22ef3a145-metrics-tls\") pod \"dns-default-lcdlg\" (UID: \"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145\") " pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:26.025607 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:26.025522    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls\") pod \"image-registry-7564dc7d49-qtdkm\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") " pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:26.025607 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:26.025595    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a173095c-7c94-4533-8529-81b464886f05-metrics-certs\") pod \"network-metrics-daemon-phwfk\" (UID: \"a173095c-7c94-4533-8529-81b464886f05\") " pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:05:26.025685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:26.025653    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls\") pod \"image-registry-668bbfbdd4-9m8pn\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") " pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:26.047328 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:26.047290    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:26.057528 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:26.057500    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-phwfk"
Nov 03 00:05:26.060419 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:26.060371    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:26.061696 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:26.061675    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:26.371140 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:26.371110    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-668bbfbdd4-9m8pn"]
Nov 03 00:05:26.379029 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:26.379006    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7b91ff_203c_4e03_9ad7_9b60215b61b6.slice/crio-afaaab5f452bf57897199a446280093b9067dd442fee1aa9f714270804e84f8b WatchSource:0}: Error finding container afaaab5f452bf57897199a446280093b9067dd442fee1aa9f714270804e84f8b: Status 404 returned error can't find the container with id afaaab5f452bf57897199a446280093b9067dd442fee1aa9f714270804e84f8b
Nov 03 00:05:26.385461 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:26.385440    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-phwfk"]
Nov 03 00:05:26.388221 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:26.388198    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda173095c_7c94_4533_8529_81b464886f05.slice/crio-9026bbbd64ae12eb7266e92016629339950102970b2a2ab44e661a97f51b6e07 WatchSource:0}: Error finding container 9026bbbd64ae12eb7266e92016629339950102970b2a2ab44e661a97f51b6e07: Status 404 returned error can't find the container with id 9026bbbd64ae12eb7266e92016629339950102970b2a2ab44e661a97f51b6e07
Nov 03 00:05:26.400404 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:26.400367    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7564dc7d49-qtdkm"]
Nov 03 00:05:26.404243 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:26.404224    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lcdlg"]
Nov 03 00:05:26.406338 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:26.406319    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7c1cb9_f311_4223_9fc7_ab4c9b8732d3.slice/crio-e0fe60209d487a7d6a73a5ea45eca3860e887d48d28d085941db33b19f0713cd WatchSource:0}: Error finding container e0fe60209d487a7d6a73a5ea45eca3860e887d48d28d085941db33b19f0713cd: Status 404 returned error can't find the container with id e0fe60209d487a7d6a73a5ea45eca3860e887d48d28d085941db33b19f0713cd
Nov 03 00:05:26.406710 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:26.406627    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cdb4c55_92a6_4791_8ab8_fcb22ef3a145.slice/crio-a56fa4e2258cf1d12ea4f7297f99b9334bc3ffd719a01a7255514c10881ffee4 WatchSource:0}: Error finding container a56fa4e2258cf1d12ea4f7297f99b9334bc3ffd719a01a7255514c10881ffee4: Status 404 returned error can't find the container with id a56fa4e2258cf1d12ea4f7297f99b9334bc3ffd719a01a7255514c10881ffee4
Nov 03 00:05:26.535071 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:26.535037    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:26.535299 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:26.535283    5012 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 00:05:27.348428 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:27.348387    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lcdlg" event={"ID":"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145","Type":"ContainerStarted","Data":"a56fa4e2258cf1d12ea4f7297f99b9334bc3ffd719a01a7255514c10881ffee4"}
Nov 03 00:05:27.350288 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:27.350253    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn" event={"ID":"4d7b91ff-203c-4e03-9ad7-9b60215b61b6","Type":"ContainerStarted","Data":"d02636a885cbe474e013db8678196280d12a8e653698338bf9ac7369d5cbfb7b"}
Nov 03 00:05:27.350443 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:27.350309    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn" event={"ID":"4d7b91ff-203c-4e03-9ad7-9b60215b61b6","Type":"ContainerStarted","Data":"afaaab5f452bf57897199a446280093b9067dd442fee1aa9f714270804e84f8b"}
Nov 03 00:05:27.353151 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:27.353122    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm" event={"ID":"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3","Type":"ContainerStarted","Data":"c2774d24b3f82251a5088a1b4f3188be261b508db5125e2d9491a32097c715ad"}
Nov 03 00:05:27.353269 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:27.353163    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm" event={"ID":"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3","Type":"ContainerStarted","Data":"e0fe60209d487a7d6a73a5ea45eca3860e887d48d28d085941db33b19f0713cd"}
Nov 03 00:05:27.354715 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:27.354677    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-phwfk" event={"ID":"a173095c-7c94-4533-8529-81b464886f05","Type":"ContainerStarted","Data":"9026bbbd64ae12eb7266e92016629339950102970b2a2ab44e661a97f51b6e07"}
Nov 03 00:05:28.359579 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:28.359530    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-phwfk" event={"ID":"a173095c-7c94-4533-8529-81b464886f05","Type":"ContainerStarted","Data":"bfd395c0f0dce064c686440b4fef493740461046db3369e76e2add66f62335f0"}
Nov 03 00:05:28.360064 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:28.359637    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-phwfk" event={"ID":"a173095c-7c94-4533-8529-81b464886f05","Type":"ContainerStarted","Data":"36181d6e4fe43659b860ae3e7c5526035c96d2d4fefeceb3e6279d0416c4207d"}
Nov 03 00:05:28.361283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:28.361251    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lcdlg" event={"ID":"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145","Type":"ContainerStarted","Data":"060098885d219b54d45eb60a40da5bb7adc8300cb3abf02eea794b3e640332ba"}
Nov 03 00:05:28.361383 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:28.361287    5012 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 00:05:28.361383 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:28.361302    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lcdlg" event={"ID":"7cdb4c55-92a6-4791-8ab8-fcb22ef3a145","Type":"ContainerStarted","Data":"fad5af7fb88d6331898b4714d792ba035c7c5a381e0d829af1d28a1beddaaa99"}
Nov 03 00:05:28.361383 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:28.361312    5012 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 00:05:28.361383 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:28.361376    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:29.236802 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:29.236771    5012 scope.go:122] "RemoveContainer" containerID="a3f3e8a79c468bc7697b87509d573d8d367915fe57bb7f44c96c4ec1b50d380c"
Nov 03 00:05:29.367410 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:29.367381    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 00:05:29.367784 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:29.367498    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d" event={"ID":"c1de1047-c4c9-4d0b-8f72-bafc79987c2e","Type":"ContainerStarted","Data":"940571c6d7e089d0eb64b4b5a9d672847dfc4bcf346e3f54e7a5033796af07f9"}
Nov 03 00:05:32.296209 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:32.296170    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:32.873035 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:32.872998    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-668bbfbdd4-9m8pn"]
Nov 03 00:05:32.873255 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:32.873243    5012 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 00:05:33.694535 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:33.694492    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:34.098652 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:34.098614    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:34.098823 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:34.098675    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2m6bp\" (UID: \"bb5193c7-10f4-4436-9c69-20e6087439c2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2m6bp"
Nov 03 00:05:34.099338 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:34.099309    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12eb002e-9765-4443-b488-6cec02b9c961-service-ca-bundle\") pod \"router-default-7df847f57-khf2f\" (UID: \"12eb002e-9765-4443-b488-6cec02b9c961\") " pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:34.101287 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:34.101258    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb5193c7-10f4-4436-9c69-20e6087439c2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2m6bp\" (UID: \"bb5193c7-10f4-4436-9c69-20e6087439c2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2m6bp"
Nov 03 00:05:34.148205 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:34.148162    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:34.151338 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:34.151305    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2m6bp"
Nov 03 00:05:34.483644 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:34.483543    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2m6bp"]
Nov 03 00:05:34.487000 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:34.486903    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5193c7_10f4_4436_9c69_20e6087439c2.slice/crio-57392cc5883b0de1d4b5d92fed6920bc6f1835a4a26b18a32efcea5af5cee9d0 WatchSource:0}: Error finding container 57392cc5883b0de1d4b5d92fed6920bc6f1835a4a26b18a32efcea5af5cee9d0: Status 404 returned error can't find the container with id 57392cc5883b0de1d4b5d92fed6920bc6f1835a4a26b18a32efcea5af5cee9d0
Nov 03 00:05:34.489214 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:34.489189    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12eb002e_9765_4443_b488_6cec02b9c961.slice/crio-0e1ac0c6d8414f18024cc8774e05bb967649c79ff10a260927e235470f68a5bc WatchSource:0}: Error finding container 0e1ac0c6d8414f18024cc8774e05bb967649c79ff10a260927e235470f68a5bc: Status 404 returned error can't find the container with id 0e1ac0c6d8414f18024cc8774e05bb967649c79ff10a260927e235470f68a5bc
Nov 03 00:05:34.492676 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:34.492649    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7df847f57-khf2f"]
Nov 03 00:05:35.392449 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:35.392411    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-2m6bp" event={"ID":"bb5193c7-10f4-4436-9c69-20e6087439c2","Type":"ContainerStarted","Data":"57392cc5883b0de1d4b5d92fed6920bc6f1835a4a26b18a32efcea5af5cee9d0"}
Nov 03 00:05:35.394025 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:35.393998    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7df847f57-khf2f" event={"ID":"12eb002e-9765-4443-b488-6cec02b9c961","Type":"ContainerStarted","Data":"d01aeb57bad2579234753af7cfbcb66fea306983d0047829c76d659e5daa8c1b"}
Nov 03 00:05:35.394144 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:35.394044    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7df847f57-khf2f" event={"ID":"12eb002e-9765-4443-b488-6cec02b9c961","Type":"ContainerStarted","Data":"0e1ac0c6d8414f18024cc8774e05bb967649c79ff10a260927e235470f68a5bc"}
Nov 03 00:05:36.149046 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:36.149017    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:36.151556 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:36.151533    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:36.400023 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:36.399928    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-2m6bp" event={"ID":"bb5193c7-10f4-4436-9c69-20e6087439c2","Type":"ContainerStarted","Data":"00cc058b60ab774e644a002d812d8519b792c01ec9818d9aea4e841853c3b4c5"}
Nov 03 00:05:36.400444 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:36.400222    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:36.401357 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:36.401336    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7df847f57-khf2f"
Nov 03 00:05:37.363069 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:37.363033    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:05:38.370089 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:38.370057    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lcdlg"
Nov 03 00:05:38.938479 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:38.938451    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ld685"
Nov 03 00:05:39.368735 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:39.368709    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:39.372493 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:39.372471    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5d95d"
Nov 03 00:05:40.253417 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.253380    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nlbnj"]
Nov 03 00:05:40.258803 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.258780    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.279802 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.279777    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 00:05:40.296874 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.296856    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 00:05:40.308960 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.308936    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-56fpg\""
Nov 03 00:05:40.308960 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.308944    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 00:05:40.309634 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.309616    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 00:05:40.448853 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.448827    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-root\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.449167 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.448857    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-metrics-client-ca\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.449167 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.448887    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-node-exporter-tls\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.449167 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.449026    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-node-exporter-accelerators-collector-config\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.449167 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.449054    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-sys\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.449167 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.449126    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.449167 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.449149    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-node-exporter-textfile\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.449385 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.449173    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-node-exporter-wtmp\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.449385 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.449220    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtbww\" (UniqueName: \"kubernetes.io/projected/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-kube-api-access-wtbww\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.550141 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.550076    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.550141 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.550109    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-node-exporter-textfile\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.550141 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.550133    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-node-exporter-wtmp\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.550401 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.550173    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtbww\" (UniqueName: \"kubernetes.io/projected/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-kube-api-access-wtbww\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.550401 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.550201    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-root\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.550401 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.550225    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-metrics-client-ca\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.550401 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.550285    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-root\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.550401 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.550322    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-node-exporter-tls\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.550401 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.550366    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-node-exporter-accelerators-collector-config\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.550401 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.550370    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-node-exporter-wtmp\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.550749 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.550405    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-sys\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.550749 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.550496    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-sys\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.550749 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.550498    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-node-exporter-textfile\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.550999 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.550980    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-metrics-client-ca\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.551229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.551210    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-node-exporter-accelerators-collector-config\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.552462 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.552441    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.552696 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.552676    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-node-exporter-tls\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.566686 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.566662    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtbww\" (UniqueName: \"kubernetes.io/projected/0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5-kube-api-access-wtbww\") pod \"node-exporter-nlbnj\" (UID: \"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5\") " pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.574537 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:40.574514    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nlbnj"
Nov 03 00:05:40.586697 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:05:40.586673    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad0a0bf_abdc_43d2_b3cb_35f9b7fee8e5.slice/crio-d39b58cb563f28c519ad382ac0209b8008402f41871eb81134e1892150473874 WatchSource:0}: Error finding container d39b58cb563f28c519ad382ac0209b8008402f41871eb81134e1892150473874: Status 404 returned error can't find the container with id d39b58cb563f28c519ad382ac0209b8008402f41871eb81134e1892150473874
Nov 03 00:05:41.420740 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:41.420698    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nlbnj" event={"ID":"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5","Type":"ContainerStarted","Data":"d39b58cb563f28c519ad382ac0209b8008402f41871eb81134e1892150473874"}
Nov 03 00:05:42.301168 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:42.301133    5012 patch_prober.go:29] interesting pod/image-registry-7564dc7d49-qtdkm container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 03 00:05:42.301560 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:42.301187    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm" podUID="ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:05:42.426782 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:42.426752    5012 generic.go:356] "Generic (PLEG): container finished" podID="0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5" containerID="b31de7ca6210615196f4ab985ef2b107acdc16dd7c8b6712e50bdc4b3a2ea44a" exitCode=0
Nov 03 00:05:42.426931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:42.426787    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nlbnj" event={"ID":"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5","Type":"ContainerDied","Data":"b31de7ca6210615196f4ab985ef2b107acdc16dd7c8b6712e50bdc4b3a2ea44a"}
Nov 03 00:05:43.432874 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:43.432839    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nlbnj" event={"ID":"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5","Type":"ContainerStarted","Data":"1759b07423716f27d8b6e9829f65b0506b31d25777ab75885592c227e2526b64"}
Nov 03 00:05:43.433295 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:43.432890    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nlbnj" event={"ID":"0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5","Type":"ContainerStarted","Data":"12d3b27aca853166f066ef56836f692f69b06c2c5a6e5ce1824676a5be421051"}
Nov 03 00:05:43.479627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:43.479580    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nlbnj" podStartSLOduration=2.5166589630000002 podStartE2EDuration="3.479567804s" totalImagesPullingTime="898.506358ms" totalInitContainerRuntime="64.402483ms" isStatefulPod=true podCreationTimestamp="2026-11-03 00:05:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:43.479396069 +0000 UTC m=+26.358980112" watchObservedRunningTime="2026-11-03 00:05:43.479567804 +0000 UTC m=+26.359151844"
Nov 03 00:05:43.699963 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:43.699872    5012 patch_prober.go:29] interesting pod/image-registry-668bbfbdd4-9m8pn container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 03 00:05:43.699963 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:43.699941    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn" podUID="4d7b91ff-203c-4e03-9ad7-9b60215b61b6" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:05:44.857509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:44.857479    5012 patch_prober.go:29] interesting pod/image-registry-7564dc7d49-qtdkm container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 03 00:05:44.857875 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:44.857527    5012 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm" podUID="ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:05:46.780769 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:46.780742    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xbrpr"
Nov 03 00:05:49.799219 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:49.799186    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ttc95"
Nov 03 00:05:52.300490 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:52.300458    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:05:53.698954 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:53.698902    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:55.007772 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:55.007743    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7564dc7d49-qtdkm"]
Nov 03 00:05:57.894100 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:57.894014    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn" podUID="4d7b91ff-203c-4e03-9ad7-9b60215b61b6" containerName="registry" containerID="cri-o://d02636a885cbe474e013db8678196280d12a8e653698338bf9ac7369d5cbfb7b" gracePeriod=30
Nov 03 00:05:58.166139 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.166078    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:58.278275 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.278241    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-kube-api-access-bbzll\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-kube-api-access-bbzll\") pod \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") "
Nov 03 00:05:58.278446 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.278287    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-image-registry-private-configuration\") pod \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") "
Nov 03 00:05:58.278446 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.278321    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-bound-sa-token\") pod \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") "
Nov 03 00:05:58.278446 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.278372    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-certificates\") pod \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") "
Nov 03 00:05:58.278446 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.278412    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls\") pod \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") "
Nov 03 00:05:58.278685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.278449    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-trusted-ca\") pod \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") "
Nov 03 00:05:58.278685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.278509    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-ca-trust-extracted\") pod \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") "
Nov 03 00:05:58.278685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.278556    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-installation-pull-secrets\") pod \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\" (UID: \"4d7b91ff-203c-4e03-9ad7-9b60215b61b6\") "
Nov 03 00:05:58.278855 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.278830    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-certificates" pod "4d7b91ff-203c-4e03-9ad7-9b60215b61b6" (UID: "4d7b91ff-203c-4e03-9ad7-9b60215b61b6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:05:58.279187 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.279139    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-trusted-ca" pod "4d7b91ff-203c-4e03-9ad7-9b60215b61b6" (UID: "4d7b91ff-203c-4e03-9ad7-9b60215b61b6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:05:58.281207 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.281173    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls" pod "4d7b91ff-203c-4e03-9ad7-9b60215b61b6" (UID: "4d7b91ff-203c-4e03-9ad7-9b60215b61b6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:05:58.281333 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.281311    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-image-registry-private-configuration" pod "4d7b91ff-203c-4e03-9ad7-9b60215b61b6" (UID: "4d7b91ff-203c-4e03-9ad7-9b60215b61b6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:05:58.281432 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.281366    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-bound-sa-token" pod "4d7b91ff-203c-4e03-9ad7-9b60215b61b6" (UID: "4d7b91ff-203c-4e03-9ad7-9b60215b61b6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:05:58.281432 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.281385    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-installation-pull-secrets" pod "4d7b91ff-203c-4e03-9ad7-9b60215b61b6" (UID: "4d7b91ff-203c-4e03-9ad7-9b60215b61b6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:05:58.281507 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.281441    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-kube-api-access-bbzll" pod "4d7b91ff-203c-4e03-9ad7-9b60215b61b6" (UID: "4d7b91ff-203c-4e03-9ad7-9b60215b61b6"). InnerVolumeSpecName "kube-api-access-bbzll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:05:58.285791 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.285726    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-ca-trust-extracted" pod "4d7b91ff-203c-4e03-9ad7-9b60215b61b6" (UID: "4d7b91ff-203c-4e03-9ad7-9b60215b61b6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:05:58.379529 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.379499    5012 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-ca-trust-extracted\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:05:58.379529 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.379528    5012 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-installation-pull-secrets\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:05:58.379699 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.379538    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbzll\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-kube-api-access-bbzll\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:05:58.379699 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.379548    5012 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-image-registry-private-configuration\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:05:58.379699 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.379559    5012 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-bound-sa-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:05:58.379699 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.379569    5012 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-certificates\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:05:58.379699 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.379577    5012 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-registry-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:05:58.379699 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.379586    5012 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4d7b91ff-203c-4e03-9ad7-9b60215b61b6-trusted-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:05:58.488328 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.488259    5012 generic.go:356] "Generic (PLEG): container finished" podID="4d7b91ff-203c-4e03-9ad7-9b60215b61b6" containerID="d02636a885cbe474e013db8678196280d12a8e653698338bf9ac7369d5cbfb7b" exitCode=0
Nov 03 00:05:58.488328 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.488307    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn"
Nov 03 00:05:58.488328 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.488313    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn" event={"ID":"4d7b91ff-203c-4e03-9ad7-9b60215b61b6","Type":"ContainerDied","Data":"d02636a885cbe474e013db8678196280d12a8e653698338bf9ac7369d5cbfb7b"}
Nov 03 00:05:58.488513 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.488353    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-668bbfbdd4-9m8pn" event={"ID":"4d7b91ff-203c-4e03-9ad7-9b60215b61b6","Type":"ContainerDied","Data":"afaaab5f452bf57897199a446280093b9067dd442fee1aa9f714270804e84f8b"}
Nov 03 00:05:58.488513 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.488372    5012 scope.go:122] "RemoveContainer" containerID="d02636a885cbe474e013db8678196280d12a8e653698338bf9ac7369d5cbfb7b"
Nov 03 00:05:58.497312 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.497292    5012 scope.go:122] "RemoveContainer" containerID="d02636a885cbe474e013db8678196280d12a8e653698338bf9ac7369d5cbfb7b"
Nov 03 00:05:58.497578 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:05:58.497557    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d02636a885cbe474e013db8678196280d12a8e653698338bf9ac7369d5cbfb7b\": container with ID starting with d02636a885cbe474e013db8678196280d12a8e653698338bf9ac7369d5cbfb7b not found: ID does not exist" containerID="d02636a885cbe474e013db8678196280d12a8e653698338bf9ac7369d5cbfb7b"
Nov 03 00:05:58.497640 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.497585    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d02636a885cbe474e013db8678196280d12a8e653698338bf9ac7369d5cbfb7b"} err="failed to get container status \"d02636a885cbe474e013db8678196280d12a8e653698338bf9ac7369d5cbfb7b\": rpc error: code = NotFound desc = could not find container \"d02636a885cbe474e013db8678196280d12a8e653698338bf9ac7369d5cbfb7b\": container with ID starting with d02636a885cbe474e013db8678196280d12a8e653698338bf9ac7369d5cbfb7b not found: ID does not exist"
Nov 03 00:05:58.533037 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.533013    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-668bbfbdd4-9m8pn"]
Nov 03 00:05:58.543007 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:58.542987    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-668bbfbdd4-9m8pn"]
Nov 03 00:05:59.241304 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:05:59.241274    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d7b91ff-203c-4e03-9ad7-9b60215b61b6" path="/var/lib/kubelet/pods/4d7b91ff-203c-4e03-9ad7-9b60215b61b6/volumes"
Nov 03 00:06:07.362300 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:06:07.362259    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:06:20.027100 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.027033    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm" podUID="ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3" containerName="registry" containerID="cri-o://c2774d24b3f82251a5088a1b4f3188be261b508db5125e2d9491a32097c715ad" gracePeriod=30
Nov 03 00:06:20.280819 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.280767    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:06:20.344758 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.344733    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-kube-api-access-s2r5h\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-kube-api-access-s2r5h\") pod \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") "
Nov 03 00:06:20.344877 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.344765    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-trusted-ca\") pod \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") "
Nov 03 00:06:20.344877 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.344797    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-certificates\") pod \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") "
Nov 03 00:06:20.344994 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.344948    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-bound-sa-token\") pod \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") "
Nov 03 00:06:20.345054 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.345011    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-installation-pull-secrets\") pod \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") "
Nov 03 00:06:20.345109 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.345049    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-image-registry-private-configuration\") pod \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") "
Nov 03 00:06:20.345109 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.345098    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-ca-trust-extracted\") pod \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") "
Nov 03 00:06:20.345207 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.345133    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls\") pod \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\" (UID: \"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3\") "
Nov 03 00:06:20.345207 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.345180    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-certificates" pod "ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3" (UID: "ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:06:20.345308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.345212    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-trusted-ca" pod "ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3" (UID: "ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:06:20.346110 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.345505    5012 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-trusted-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:06:20.346110 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.345529    5012 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-certificates\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:06:20.347655 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.347606    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-bound-sa-token" pod "ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3" (UID: "ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:06:20.347785 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.347747    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-image-registry-private-configuration" pod "ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3" (UID: "ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:06:20.347853 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.347790    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls" pod "ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3" (UID: "ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:06:20.347853 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.347796    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-kube-api-access-s2r5h" pod "ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3" (UID: "ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3"). InnerVolumeSpecName "kube-api-access-s2r5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:06:20.348060 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.348026    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-installation-pull-secrets" pod "ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3" (UID: "ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:06:20.352030 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.352002    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-ca-trust-extracted" pod "ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3" (UID: "ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:06:20.445780 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.445760    5012 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-bound-sa-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:06:20.445780 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.445779    5012 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-installation-pull-secrets\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:06:20.445901 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.445791    5012 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-image-registry-private-configuration\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:06:20.445901 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.445801    5012 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-ca-trust-extracted\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:06:20.445901 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.445811    5012 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-registry-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:06:20.445901 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.445821    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2r5h\" (UniqueName: \"kubernetes.io/projected/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3-kube-api-access-s2r5h\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:06:20.571863 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.571807    5012 generic.go:356] "Generic (PLEG): container finished" podID="ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3" containerID="c2774d24b3f82251a5088a1b4f3188be261b508db5125e2d9491a32097c715ad" exitCode=0
Nov 03 00:06:20.571863 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.571855    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm"
Nov 03 00:06:20.572049 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.571897    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm" event={"ID":"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3","Type":"ContainerDied","Data":"c2774d24b3f82251a5088a1b4f3188be261b508db5125e2d9491a32097c715ad"}
Nov 03 00:06:20.572049 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.571977    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7564dc7d49-qtdkm" event={"ID":"ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3","Type":"ContainerDied","Data":"e0fe60209d487a7d6a73a5ea45eca3860e887d48d28d085941db33b19f0713cd"}
Nov 03 00:06:20.572049 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.572005    5012 scope.go:122] "RemoveContainer" containerID="c2774d24b3f82251a5088a1b4f3188be261b508db5125e2d9491a32097c715ad"
Nov 03 00:06:20.580709 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.580695    5012 scope.go:122] "RemoveContainer" containerID="c2774d24b3f82251a5088a1b4f3188be261b508db5125e2d9491a32097c715ad"
Nov 03 00:06:20.580995 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:06:20.580971    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2774d24b3f82251a5088a1b4f3188be261b508db5125e2d9491a32097c715ad\": container with ID starting with c2774d24b3f82251a5088a1b4f3188be261b508db5125e2d9491a32097c715ad not found: ID does not exist" containerID="c2774d24b3f82251a5088a1b4f3188be261b508db5125e2d9491a32097c715ad"
Nov 03 00:06:20.581084 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.581001    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2774d24b3f82251a5088a1b4f3188be261b508db5125e2d9491a32097c715ad"} err="failed to get container status \"c2774d24b3f82251a5088a1b4f3188be261b508db5125e2d9491a32097c715ad\": rpc error: code = NotFound desc = could not find container \"c2774d24b3f82251a5088a1b4f3188be261b508db5125e2d9491a32097c715ad\": container with ID starting with c2774d24b3f82251a5088a1b4f3188be261b508db5125e2d9491a32097c715ad not found: ID does not exist"
Nov 03 00:06:20.614400 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.614382    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7564dc7d49-qtdkm"]
Nov 03 00:06:20.624607 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:20.624588    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7564dc7d49-qtdkm"]
Nov 03 00:06:21.241063 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:21.241030    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3" path="/var/lib/kubelet/pods/ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3/volumes"
Nov 03 00:06:27.600043 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:27.600018    5012 generic.go:356] "Generic (PLEG): container finished" podID="38b66660-5608-4e82-a1eb-db3d6ce57ca5" containerID="e3357bdcf4cbec9868e9d50e392c3a72be4ee9619f5333683d160c5d649df639" exitCode=0
Nov 03 00:06:27.600395 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:27.600091    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc" event={"ID":"38b66660-5608-4e82-a1eb-db3d6ce57ca5","Type":"ContainerDied","Data":"e3357bdcf4cbec9868e9d50e392c3a72be4ee9619f5333683d160c5d649df639"}
Nov 03 00:06:27.600395 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:27.600340    5012 scope.go:122] "RemoveContainer" containerID="e3357bdcf4cbec9868e9d50e392c3a72be4ee9619f5333683d160c5d649df639"
Nov 03 00:06:28.607489 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:28.607444    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tqwsc" event={"ID":"38b66660-5608-4e82-a1eb-db3d6ce57ca5","Type":"ContainerStarted","Data":"72526e20fac122454fc65acfce00f161a21e9633cd65c0b6712d1442d92505a4"}
Nov 03 00:06:37.361245 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:06:37.361214    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:06:47.682650 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:47.682613    5012 generic.go:356] "Generic (PLEG): container finished" podID="a0d4ed33-0500-4c65-ba72-b2a584b6bdf2" containerID="4d5e7453f09dfd64250c0cd6b64caf6ebc2043881d9acba0d3b880b12da8d44a" exitCode=0
Nov 03 00:06:47.683158 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:47.682691    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-kzsm6" event={"ID":"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2","Type":"ContainerDied","Data":"4d5e7453f09dfd64250c0cd6b64caf6ebc2043881d9acba0d3b880b12da8d44a"}
Nov 03 00:06:47.683158 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:47.683138    5012 scope.go:122] "RemoveContainer" containerID="4d5e7453f09dfd64250c0cd6b64caf6ebc2043881d9acba0d3b880b12da8d44a"
Nov 03 00:06:48.258032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.257992    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-w25g4-jgbzw"]
Nov 03 00:06:48.258312 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.258298    5012 cpu_manager.go:414] "containerMap: removing container" podUID="ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3" containerName="registry"
Nov 03 00:06:48.258362 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.258313    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3" containerName="registry"
Nov 03 00:06:48.258362 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.258326    5012 cpu_manager.go:414] "containerMap: removing container" podUID="4d7b91ff-203c-4e03-9ad7-9b60215b61b6" containerName="registry"
Nov 03 00:06:48.258362 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.258332    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d7b91ff-203c-4e03-9ad7-9b60215b61b6" containerName="registry"
Nov 03 00:06:48.258460 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.258387    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed7c1cb9-f311-4223-9fc7-ab4c9b8732d3" containerName="registry"
Nov 03 00:06:48.258460 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.258396    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d7b91ff-203c-4e03-9ad7-9b60215b61b6" containerName="registry"
Nov 03 00:06:48.261367 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.261339    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-w25g4-jgbzw"
Nov 03 00:06:48.276403 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.276372    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-w25g4-jgbzw"]
Nov 03 00:06:48.335168 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.335131    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv485\" (UniqueName: \"kubernetes.io/projected/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-kube-api-access-xv485\") pod \"periodic-gathering-w25g4-jgbzw\" (UID: \"3ee5dc12-a42b-4121-ba8f-259a0d04a94e\") " pod="openshift-insights/periodic-gathering-w25g4-jgbzw"
Nov 03 00:06:48.335168 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.335166    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-archives-path\") pod \"periodic-gathering-w25g4-jgbzw\" (UID: \"3ee5dc12-a42b-4121-ba8f-259a0d04a94e\") " pod="openshift-insights/periodic-gathering-w25g4-jgbzw"
Nov 03 00:06:48.335384 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.335200    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-service-ca-bundle\") pod \"periodic-gathering-w25g4-jgbzw\" (UID: \"3ee5dc12-a42b-4121-ba8f-259a0d04a94e\") " pod="openshift-insights/periodic-gathering-w25g4-jgbzw"
Nov 03 00:06:48.335384 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.335364    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-trusted-ca-bundle\") pod \"periodic-gathering-w25g4-jgbzw\" (UID: \"3ee5dc12-a42b-4121-ba8f-259a0d04a94e\") " pod="openshift-insights/periodic-gathering-w25g4-jgbzw"
Nov 03 00:06:48.436145 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.436103    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-service-ca-bundle\") pod \"periodic-gathering-w25g4-jgbzw\" (UID: \"3ee5dc12-a42b-4121-ba8f-259a0d04a94e\") " pod="openshift-insights/periodic-gathering-w25g4-jgbzw"
Nov 03 00:06:48.436328 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.436161    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-trusted-ca-bundle\") pod \"periodic-gathering-w25g4-jgbzw\" (UID: \"3ee5dc12-a42b-4121-ba8f-259a0d04a94e\") " pod="openshift-insights/periodic-gathering-w25g4-jgbzw"
Nov 03 00:06:48.436328 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.436189    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv485\" (UniqueName: \"kubernetes.io/projected/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-kube-api-access-xv485\") pod \"periodic-gathering-w25g4-jgbzw\" (UID: \"3ee5dc12-a42b-4121-ba8f-259a0d04a94e\") " pod="openshift-insights/periodic-gathering-w25g4-jgbzw"
Nov 03 00:06:48.436328 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.436212    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-archives-path\") pod \"periodic-gathering-w25g4-jgbzw\" (UID: \"3ee5dc12-a42b-4121-ba8f-259a0d04a94e\") " pod="openshift-insights/periodic-gathering-w25g4-jgbzw"
Nov 03 00:06:48.436645 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.436617    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-archives-path\") pod \"periodic-gathering-w25g4-jgbzw\" (UID: \"3ee5dc12-a42b-4121-ba8f-259a0d04a94e\") " pod="openshift-insights/periodic-gathering-w25g4-jgbzw"
Nov 03 00:06:48.436807 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.436789    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-service-ca-bundle\") pod \"periodic-gathering-w25g4-jgbzw\" (UID: \"3ee5dc12-a42b-4121-ba8f-259a0d04a94e\") " pod="openshift-insights/periodic-gathering-w25g4-jgbzw"
Nov 03 00:06:48.437054 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.437037    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-trusted-ca-bundle\") pod \"periodic-gathering-w25g4-jgbzw\" (UID: \"3ee5dc12-a42b-4121-ba8f-259a0d04a94e\") " pod="openshift-insights/periodic-gathering-w25g4-jgbzw"
Nov 03 00:06:48.452973 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.452947    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv485\" (UniqueName: \"kubernetes.io/projected/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-kube-api-access-xv485\") pod \"periodic-gathering-w25g4-jgbzw\" (UID: \"3ee5dc12-a42b-4121-ba8f-259a0d04a94e\") " pod="openshift-insights/periodic-gathering-w25g4-jgbzw"
Nov 03 00:06:48.572823 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.572698    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-w25g4-jgbzw"
Nov 03 00:06:48.688276 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.688236    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-kzsm6" event={"ID":"a0d4ed33-0500-4c65-ba72-b2a584b6bdf2","Type":"ContainerStarted","Data":"b22eae622a69353d04bdb5d4ff92580653bd4531a7cf570abe43436b5e0062f4"}
Nov 03 00:06:48.689728 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.689702    5012 generic.go:356] "Generic (PLEG): container finished" podID="689b30e8-936e-49a4-a775-4ce908ad1ab8" containerID="e83ccef58b1c16a55c030ca8700573bcba68bd1ef3943055a38bff4bbfe75a38" exitCode=0
Nov 03 00:06:48.689840 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.689753    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p" event={"ID":"689b30e8-936e-49a4-a775-4ce908ad1ab8","Type":"ContainerDied","Data":"e83ccef58b1c16a55c030ca8700573bcba68bd1ef3943055a38bff4bbfe75a38"}
Nov 03 00:06:48.690069 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.690055    5012 scope.go:122] "RemoveContainer" containerID="e83ccef58b1c16a55c030ca8700573bcba68bd1ef3943055a38bff4bbfe75a38"
Nov 03 00:06:48.862688 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:48.862597    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-w25g4-jgbzw"]
Nov 03 00:06:48.865411 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:06:48.865385    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee5dc12_a42b_4121_ba8f_259a0d04a94e.slice/crio-b89ece7f8aaa9770628a93d6b391f601093e4658db1d5bfc506182ef989b5590 WatchSource:0}: Error finding container b89ece7f8aaa9770628a93d6b391f601093e4658db1d5bfc506182ef989b5590: Status 404 returned error can't find the container with id b89ece7f8aaa9770628a93d6b391f601093e4658db1d5bfc506182ef989b5590
Nov 03 00:06:49.695161 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:49.695127    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-w25g4-jgbzw" event={"ID":"3ee5dc12-a42b-4121-ba8f-259a0d04a94e","Type":"ContainerStarted","Data":"ee8215cb10f1d91096847519aa6a86d15807c31487118d79c13e7ab7e9fe2204"}
Nov 03 00:06:49.695562 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:49.695184    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-w25g4-jgbzw" event={"ID":"3ee5dc12-a42b-4121-ba8f-259a0d04a94e","Type":"ContainerStarted","Data":"b89ece7f8aaa9770628a93d6b391f601093e4658db1d5bfc506182ef989b5590"}
Nov 03 00:06:49.697016 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:49.696994    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-w6k5p" event={"ID":"689b30e8-936e-49a4-a775-4ce908ad1ab8","Type":"ContainerStarted","Data":"bcf02b1f46048da4edffc10ef2c8aca7cfe6a6058d9c8e96a04aae28742f5df0"}
Nov 03 00:06:49.726755 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:06:49.726710    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-w25g4-jgbzw" podStartSLOduration=1.726699541 podStartE2EDuration="1.726699541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:06:49.72643017 +0000 UTC m=+92.606014213" watchObservedRunningTime="2026-11-03 00:06:49.726699541 +0000 UTC m=+92.606283581"
Nov 03 00:07:07.363952 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:07:07.363895    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:07:37.362123 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:07:37.362040    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:08:07.361146 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:08:07.361111    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:08:09.005194 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:09.005167    5012 generic.go:356] "Generic (PLEG): container finished" podID="daa3da9a-3a5f-49a4-8a09-0036fdc5aa44" containerID="c31fe2c21e01834528f35f724afb86b8577a5b769c8caf26daa715d4c4c00c0a" exitCode=0
Nov 03 00:08:09.005495 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:09.005244    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29894400-s5xp5" event={"ID":"daa3da9a-3a5f-49a4-8a09-0036fdc5aa44","Type":"ContainerDied","Data":"c31fe2c21e01834528f35f724afb86b8577a5b769c8caf26daa715d4c4c00c0a"}
Nov 03 00:08:10.135290 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:10.135266    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29894400-s5xp5"
Nov 03 00:08:10.236484 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:10.236453    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daa3da9a-3a5f-49a4-8a09-0036fdc5aa44-kube-api-access-kscjb\" (UniqueName: \"kubernetes.io/projected/daa3da9a-3a5f-49a4-8a09-0036fdc5aa44-kube-api-access-kscjb\") pod \"daa3da9a-3a5f-49a4-8a09-0036fdc5aa44\" (UID: \"daa3da9a-3a5f-49a4-8a09-0036fdc5aa44\") "
Nov 03 00:08:10.236626 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:10.236558    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daa3da9a-3a5f-49a4-8a09-0036fdc5aa44-serviceca\" (UniqueName: \"kubernetes.io/configmap/daa3da9a-3a5f-49a4-8a09-0036fdc5aa44-serviceca\") pod \"daa3da9a-3a5f-49a4-8a09-0036fdc5aa44\" (UID: \"daa3da9a-3a5f-49a4-8a09-0036fdc5aa44\") "
Nov 03 00:08:10.236897 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:10.236873    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daa3da9a-3a5f-49a4-8a09-0036fdc5aa44-serviceca" pod "daa3da9a-3a5f-49a4-8a09-0036fdc5aa44" (UID: "daa3da9a-3a5f-49a4-8a09-0036fdc5aa44"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:08:10.238624 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:10.238596    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa3da9a-3a5f-49a4-8a09-0036fdc5aa44-kube-api-access-kscjb" pod "daa3da9a-3a5f-49a4-8a09-0036fdc5aa44" (UID: "daa3da9a-3a5f-49a4-8a09-0036fdc5aa44"). InnerVolumeSpecName "kube-api-access-kscjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:08:10.337446 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:10.337395    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kscjb\" (UniqueName: \"kubernetes.io/projected/daa3da9a-3a5f-49a4-8a09-0036fdc5aa44-kube-api-access-kscjb\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:08:10.337446 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:10.337415    5012 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/daa3da9a-3a5f-49a4-8a09-0036fdc5aa44-serviceca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:08:11.014636 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:11.014597    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29894400-s5xp5" event={"ID":"daa3da9a-3a5f-49a4-8a09-0036fdc5aa44","Type":"ContainerDied","Data":"e534f8d6a73d1ff175ee0a5a2695bd877777cb30a483c171cafda7c5946ace57"}
Nov 03 00:08:11.014798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:11.014647    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e534f8d6a73d1ff175ee0a5a2695bd877777cb30a483c171cafda7c5946ace57"
Nov 03 00:08:11.014798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:11.014608    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29894400-s5xp5"
Nov 03 00:08:13.023742 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:13.023710    5012 generic.go:356] "Generic (PLEG): container finished" podID="3ee5dc12-a42b-4121-ba8f-259a0d04a94e" containerID="ee8215cb10f1d91096847519aa6a86d15807c31487118d79c13e7ab7e9fe2204" exitCode=0
Nov 03 00:08:13.024213 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:13.023782    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-w25g4-jgbzw" event={"ID":"3ee5dc12-a42b-4121-ba8f-259a0d04a94e","Type":"ContainerDied","Data":"ee8215cb10f1d91096847519aa6a86d15807c31487118d79c13e7ab7e9fe2204"}
Nov 03 00:08:14.157515 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:14.157492    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-w25g4-jgbzw"
Nov 03 00:08:14.261787 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:14.261756    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-service-ca-bundle\") pod \"3ee5dc12-a42b-4121-ba8f-259a0d04a94e\" (UID: \"3ee5dc12-a42b-4121-ba8f-259a0d04a94e\") "
Nov 03 00:08:14.261958 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:14.261816    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-trusted-ca-bundle\") pod \"3ee5dc12-a42b-4121-ba8f-259a0d04a94e\" (UID: \"3ee5dc12-a42b-4121-ba8f-259a0d04a94e\") "
Nov 03 00:08:14.262035 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:14.261987    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-kube-api-access-xv485\" (UniqueName: \"kubernetes.io/projected/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-kube-api-access-xv485\") pod \"3ee5dc12-a42b-4121-ba8f-259a0d04a94e\" (UID: \"3ee5dc12-a42b-4121-ba8f-259a0d04a94e\") "
Nov 03 00:08:14.262098 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:14.262029    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-archives-path\") pod \"3ee5dc12-a42b-4121-ba8f-259a0d04a94e\" (UID: \"3ee5dc12-a42b-4121-ba8f-259a0d04a94e\") "
Nov 03 00:08:14.262154 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:14.262099    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-service-ca-bundle" pod "3ee5dc12-a42b-4121-ba8f-259a0d04a94e" (UID: "3ee5dc12-a42b-4121-ba8f-259a0d04a94e"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:08:14.262314 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:14.262220    5012 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-service-ca-bundle\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:08:14.262373 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:14.262289    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-trusted-ca-bundle" pod "3ee5dc12-a42b-4121-ba8f-259a0d04a94e" (UID: "3ee5dc12-a42b-4121-ba8f-259a0d04a94e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:08:14.262373 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:14.262341    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-archives-path" pod "3ee5dc12-a42b-4121-ba8f-259a0d04a94e" (UID: "3ee5dc12-a42b-4121-ba8f-259a0d04a94e"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:08:14.264126 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:14.264110    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-kube-api-access-xv485" pod "3ee5dc12-a42b-4121-ba8f-259a0d04a94e" (UID: "3ee5dc12-a42b-4121-ba8f-259a0d04a94e"). InnerVolumeSpecName "kube-api-access-xv485". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:08:14.362552 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:14.362534    5012 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-trusted-ca-bundle\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:08:14.362552 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:14.362553    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv485\" (UniqueName: \"kubernetes.io/projected/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-kube-api-access-xv485\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:08:14.362696 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:14.362563    5012 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3ee5dc12-a42b-4121-ba8f-259a0d04a94e-archives-path\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:08:15.032721 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:15.032688    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-w25g4-jgbzw"
Nov 03 00:08:15.032933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:15.032713    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-w25g4-jgbzw" event={"ID":"3ee5dc12-a42b-4121-ba8f-259a0d04a94e","Type":"ContainerDied","Data":"b89ece7f8aaa9770628a93d6b391f601093e4658db1d5bfc506182ef989b5590"}
Nov 03 00:08:15.032933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:15.032766    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b89ece7f8aaa9770628a93d6b391f601093e4658db1d5bfc506182ef989b5590"
Nov 03 00:08:37.361623 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:08:37.361583    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:08:49.953700 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:49.953663    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b"]
Nov 03 00:08:49.954209 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:49.953977    5012 cpu_manager.go:414] "containerMap: removing container" podUID="daa3da9a-3a5f-49a4-8a09-0036fdc5aa44" containerName="image-pruner"
Nov 03 00:08:49.954209 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:49.953989    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daa3da9a-3a5f-49a4-8a09-0036fdc5aa44" containerName="image-pruner"
Nov 03 00:08:49.954209 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:49.953999    5012 cpu_manager.go:414] "containerMap: removing container" podUID="3ee5dc12-a42b-4121-ba8f-259a0d04a94e" containerName="insights-gathering"
Nov 03 00:08:49.954209 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:49.954004    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ee5dc12-a42b-4121-ba8f-259a0d04a94e" containerName="insights-gathering"
Nov 03 00:08:49.954209 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:49.954055    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="daa3da9a-3a5f-49a4-8a09-0036fdc5aa44" containerName="image-pruner"
Nov 03 00:08:49.954209 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:49.954062    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ee5dc12-a42b-4121-ba8f-259a0d04a94e" containerName="insights-gathering"
Nov 03 00:08:49.957275 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:49.957256    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b"
Nov 03 00:08:49.969597 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:49.969577    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 00:08:49.979475 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:49.979449    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 00:08:49.979786 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:49.979725    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-ttskr\""
Nov 03 00:08:49.980700 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:49.980672    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b"]
Nov 03 00:08:50.091185 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:50.091157    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmfv6\" (UniqueName: \"kubernetes.io/projected/0158fe4a-e132-4d0e-92e9-8ce181c2c559-kube-api-access-bmfv6\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b\" (UID: \"0158fe4a-e132-4d0e-92e9-8ce181c2c559\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b"
Nov 03 00:08:50.091286 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:50.091188    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0158fe4a-e132-4d0e-92e9-8ce181c2c559-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b\" (UID: \"0158fe4a-e132-4d0e-92e9-8ce181c2c559\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b"
Nov 03 00:08:50.091286 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:50.091217    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0158fe4a-e132-4d0e-92e9-8ce181c2c559-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b\" (UID: \"0158fe4a-e132-4d0e-92e9-8ce181c2c559\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b"
Nov 03 00:08:50.192278 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:50.192250    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0158fe4a-e132-4d0e-92e9-8ce181c2c559-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b\" (UID: \"0158fe4a-e132-4d0e-92e9-8ce181c2c559\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b"
Nov 03 00:08:50.192381 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:50.192289    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0158fe4a-e132-4d0e-92e9-8ce181c2c559-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b\" (UID: \"0158fe4a-e132-4d0e-92e9-8ce181c2c559\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b"
Nov 03 00:08:50.192381 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:50.192342    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmfv6\" (UniqueName: \"kubernetes.io/projected/0158fe4a-e132-4d0e-92e9-8ce181c2c559-kube-api-access-bmfv6\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b\" (UID: \"0158fe4a-e132-4d0e-92e9-8ce181c2c559\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b"
Nov 03 00:08:50.192630 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:50.192613    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0158fe4a-e132-4d0e-92e9-8ce181c2c559-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b\" (UID: \"0158fe4a-e132-4d0e-92e9-8ce181c2c559\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b"
Nov 03 00:08:50.192671 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:50.192660    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0158fe4a-e132-4d0e-92e9-8ce181c2c559-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b\" (UID: \"0158fe4a-e132-4d0e-92e9-8ce181c2c559\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b"
Nov 03 00:08:50.215719 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:50.215662    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmfv6\" (UniqueName: \"kubernetes.io/projected/0158fe4a-e132-4d0e-92e9-8ce181c2c559-kube-api-access-bmfv6\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b\" (UID: \"0158fe4a-e132-4d0e-92e9-8ce181c2c559\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b"
Nov 03 00:08:50.272775 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:50.272753    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b"
Nov 03 00:08:50.564460 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:50.564430    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b"]
Nov 03 00:08:50.567001 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:08:50.566966    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0158fe4a_e132_4d0e_92e9_8ce181c2c559.slice/crio-4db1f60539ef1edffced89272827b24367ce3b9c4617a5d1aea205b0b9850bc7 WatchSource:0}: Error finding container 4db1f60539ef1edffced89272827b24367ce3b9c4617a5d1aea205b0b9850bc7: Status 404 returned error can't find the container with id 4db1f60539ef1edffced89272827b24367ce3b9c4617a5d1aea205b0b9850bc7
Nov 03 00:08:51.071305 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.071275    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md"]
Nov 03 00:08:51.074847 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.074820    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md"
Nov 03 00:08:51.103622 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.103588    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md"]
Nov 03 00:08:51.170775 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.170749    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-bmcmd\""
Nov 03 00:08:51.183260 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.183230    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b" event={"ID":"0158fe4a-e132-4d0e-92e9-8ce181c2c559","Type":"ContainerStarted","Data":"4db1f60539ef1edffced89272827b24367ce3b9c4617a5d1aea205b0b9850bc7"}
Nov 03 00:08:51.199732 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.199705    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acb40932-dcac-4068-a88b-1479f7904f3c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md\" (UID: \"acb40932-dcac-4068-a88b-1479f7904f3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md"
Nov 03 00:08:51.199838 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.199745    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acb40932-dcac-4068-a88b-1479f7904f3c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md\" (UID: \"acb40932-dcac-4068-a88b-1479f7904f3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md"
Nov 03 00:08:51.199902 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.199838    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5dkt\" (UniqueName: \"kubernetes.io/projected/acb40932-dcac-4068-a88b-1479f7904f3c-kube-api-access-f5dkt\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md\" (UID: \"acb40932-dcac-4068-a88b-1479f7904f3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md"
Nov 03 00:08:51.264605 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.264573    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2"]
Nov 03 00:08:51.268119 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.268099    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2"
Nov 03 00:08:51.281088 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.281069    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-dgd27\""
Nov 03 00:08:51.287662 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.287629    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2"]
Nov 03 00:08:51.300565 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.300522    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acb40932-dcac-4068-a88b-1479f7904f3c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md\" (UID: \"acb40932-dcac-4068-a88b-1479f7904f3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md"
Nov 03 00:08:51.300677 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.300580    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acb40932-dcac-4068-a88b-1479f7904f3c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md\" (UID: \"acb40932-dcac-4068-a88b-1479f7904f3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md"
Nov 03 00:08:51.300677 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.300620    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5dkt\" (UniqueName: \"kubernetes.io/projected/acb40932-dcac-4068-a88b-1479f7904f3c-kube-api-access-f5dkt\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md\" (UID: \"acb40932-dcac-4068-a88b-1479f7904f3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md"
Nov 03 00:08:51.300951 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.300929    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acb40932-dcac-4068-a88b-1479f7904f3c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md\" (UID: \"acb40932-dcac-4068-a88b-1479f7904f3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md"
Nov 03 00:08:51.301022 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.300972    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acb40932-dcac-4068-a88b-1479f7904f3c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md\" (UID: \"acb40932-dcac-4068-a88b-1479f7904f3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md"
Nov 03 00:08:51.318348 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.318309    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5dkt\" (UniqueName: \"kubernetes.io/projected/acb40932-dcac-4068-a88b-1479f7904f3c-kube-api-access-f5dkt\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md\" (UID: \"acb40932-dcac-4068-a88b-1479f7904f3c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md"
Nov 03 00:08:51.346107 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.346051    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9"]
Nov 03 00:08:51.350093 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.350075    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9"
Nov 03 00:08:51.366101 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.366080    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-gl9gs\""
Nov 03 00:08:51.367604 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.367587    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9"]
Nov 03 00:08:51.392991 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.392960    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md"
Nov 03 00:08:51.402053 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.402025    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96e7a1a7-b210-41d5-af65-4722b7b33e67-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2\" (UID: \"96e7a1a7-b210-41d5-af65-4722b7b33e67\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2"
Nov 03 00:08:51.402211 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.402185    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdbhl\" (UniqueName: \"kubernetes.io/projected/96e7a1a7-b210-41d5-af65-4722b7b33e67-kube-api-access-kdbhl\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2\" (UID: \"96e7a1a7-b210-41d5-af65-4722b7b33e67\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2"
Nov 03 00:08:51.402281 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.402239    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96e7a1a7-b210-41d5-af65-4722b7b33e67-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2\" (UID: \"96e7a1a7-b210-41d5-af65-4722b7b33e67\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2"
Nov 03 00:08:51.444502 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.444466    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c"]
Nov 03 00:08:51.450839 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.449050    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c"
Nov 03 00:08:51.465621 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.465587    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-hf797\""
Nov 03 00:08:51.467087 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.467056    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c"]
Nov 03 00:08:51.503662 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.503515    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxfgp\" (UniqueName: \"kubernetes.io/projected/15b7bc3d-86d7-4dc5-93d5-8f213d700093-kube-api-access-bxfgp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9\" (UID: \"15b7bc3d-86d7-4dc5-93d5-8f213d700093\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9"
Nov 03 00:08:51.503662 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.503601    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15b7bc3d-86d7-4dc5-93d5-8f213d700093-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9\" (UID: \"15b7bc3d-86d7-4dc5-93d5-8f213d700093\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9"
Nov 03 00:08:51.503662 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.503635    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96e7a1a7-b210-41d5-af65-4722b7b33e67-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2\" (UID: \"96e7a1a7-b210-41d5-af65-4722b7b33e67\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2"
Nov 03 00:08:51.503974 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.503757    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15b7bc3d-86d7-4dc5-93d5-8f213d700093-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9\" (UID: \"15b7bc3d-86d7-4dc5-93d5-8f213d700093\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9"
Nov 03 00:08:51.503974 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.503852    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdbhl\" (UniqueName: \"kubernetes.io/projected/96e7a1a7-b210-41d5-af65-4722b7b33e67-kube-api-access-kdbhl\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2\" (UID: \"96e7a1a7-b210-41d5-af65-4722b7b33e67\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2"
Nov 03 00:08:51.503974 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.503903    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96e7a1a7-b210-41d5-af65-4722b7b33e67-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2\" (UID: \"96e7a1a7-b210-41d5-af65-4722b7b33e67\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2"
Nov 03 00:08:51.504159 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.504049    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96e7a1a7-b210-41d5-af65-4722b7b33e67-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2\" (UID: \"96e7a1a7-b210-41d5-af65-4722b7b33e67\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2"
Nov 03 00:08:51.504281 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.504252    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96e7a1a7-b210-41d5-af65-4722b7b33e67-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2\" (UID: \"96e7a1a7-b210-41d5-af65-4722b7b33e67\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2"
Nov 03 00:08:51.520755 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.520731    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdbhl\" (UniqueName: \"kubernetes.io/projected/96e7a1a7-b210-41d5-af65-4722b7b33e67-kube-api-access-kdbhl\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2\" (UID: \"96e7a1a7-b210-41d5-af65-4722b7b33e67\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2"
Nov 03 00:08:51.545054 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.545031    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j"]
Nov 03 00:08:51.548795 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.548771    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j"
Nov 03 00:08:51.565955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.565930    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-wm84p\""
Nov 03 00:08:51.567051 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.567030    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j"]
Nov 03 00:08:51.584613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.584593    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2"
Nov 03 00:08:51.604591 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.604564    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxfgp\" (UniqueName: \"kubernetes.io/projected/15b7bc3d-86d7-4dc5-93d5-8f213d700093-kube-api-access-bxfgp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9\" (UID: \"15b7bc3d-86d7-4dc5-93d5-8f213d700093\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9"
Nov 03 00:08:51.604687 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.604594    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15b7bc3d-86d7-4dc5-93d5-8f213d700093-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9\" (UID: \"15b7bc3d-86d7-4dc5-93d5-8f213d700093\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9"
Nov 03 00:08:51.604687 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.604617    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a276be2c-84f0-4457-91fa-d5dedacaa865-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c\" (UID: \"a276be2c-84f0-4457-91fa-d5dedacaa865\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c"
Nov 03 00:08:51.604687 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.604641    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a276be2c-84f0-4457-91fa-d5dedacaa865-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c\" (UID: \"a276be2c-84f0-4457-91fa-d5dedacaa865\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c"
Nov 03 00:08:51.604687 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.604676    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15b7bc3d-86d7-4dc5-93d5-8f213d700093-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9\" (UID: \"15b7bc3d-86d7-4dc5-93d5-8f213d700093\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9"
Nov 03 00:08:51.604849 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.604806    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnbhr\" (UniqueName: \"kubernetes.io/projected/a276be2c-84f0-4457-91fa-d5dedacaa865-kube-api-access-nnbhr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c\" (UID: \"a276be2c-84f0-4457-91fa-d5dedacaa865\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c"
Nov 03 00:08:51.605015 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.604999    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15b7bc3d-86d7-4dc5-93d5-8f213d700093-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9\" (UID: \"15b7bc3d-86d7-4dc5-93d5-8f213d700093\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9"
Nov 03 00:08:51.605143 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.605123    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15b7bc3d-86d7-4dc5-93d5-8f213d700093-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9\" (UID: \"15b7bc3d-86d7-4dc5-93d5-8f213d700093\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9"
Nov 03 00:08:51.621294 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.621268    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxfgp\" (UniqueName: \"kubernetes.io/projected/15b7bc3d-86d7-4dc5-93d5-8f213d700093-kube-api-access-bxfgp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9\" (UID: \"15b7bc3d-86d7-4dc5-93d5-8f213d700093\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9"
Nov 03 00:08:51.668362 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.668330    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9"
Nov 03 00:08:51.694866 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.694810    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md"]
Nov 03 00:08:51.697129 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:08:51.697094    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacb40932_dcac_4068_a88b_1479f7904f3c.slice/crio-f9dae51b9f74192925ac514132dcb4dcf777ee74af929013ca57c2707fcb1b7d WatchSource:0}: Error finding container f9dae51b9f74192925ac514132dcb4dcf777ee74af929013ca57c2707fcb1b7d: Status 404 returned error can't find the container with id f9dae51b9f74192925ac514132dcb4dcf777ee74af929013ca57c2707fcb1b7d
Nov 03 00:08:51.705904 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.705875    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnbhr\" (UniqueName: \"kubernetes.io/projected/a276be2c-84f0-4457-91fa-d5dedacaa865-kube-api-access-nnbhr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c\" (UID: \"a276be2c-84f0-4457-91fa-d5dedacaa865\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c"
Nov 03 00:08:51.706016 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.705948    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jpbw\" (UniqueName: \"kubernetes.io/projected/0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945-kube-api-access-6jpbw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j\" (UID: \"0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j"
Nov 03 00:08:51.706016 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.705976    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j\" (UID: \"0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j"
Nov 03 00:08:51.706016 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.705999    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a276be2c-84f0-4457-91fa-d5dedacaa865-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c\" (UID: \"a276be2c-84f0-4457-91fa-d5dedacaa865\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c"
Nov 03 00:08:51.706171 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.706016    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a276be2c-84f0-4457-91fa-d5dedacaa865-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c\" (UID: \"a276be2c-84f0-4457-91fa-d5dedacaa865\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c"
Nov 03 00:08:51.706171 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.706065    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j\" (UID: \"0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j"
Nov 03 00:08:51.706459 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.706437    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a276be2c-84f0-4457-91fa-d5dedacaa865-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c\" (UID: \"a276be2c-84f0-4457-91fa-d5dedacaa865\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c"
Nov 03 00:08:51.706541 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.706503    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a276be2c-84f0-4457-91fa-d5dedacaa865-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c\" (UID: \"a276be2c-84f0-4457-91fa-d5dedacaa865\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c"
Nov 03 00:08:51.725122 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.725097    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnbhr\" (UniqueName: \"kubernetes.io/projected/a276be2c-84f0-4457-91fa-d5dedacaa865-kube-api-access-nnbhr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c\" (UID: \"a276be2c-84f0-4457-91fa-d5dedacaa865\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c"
Nov 03 00:08:51.774478 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.774448    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c"
Nov 03 00:08:51.807000 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.806968    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j\" (UID: \"0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j"
Nov 03 00:08:51.807134 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.807019    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jpbw\" (UniqueName: \"kubernetes.io/projected/0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945-kube-api-access-6jpbw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j\" (UID: \"0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j"
Nov 03 00:08:51.807134 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.807047    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j\" (UID: \"0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j"
Nov 03 00:08:51.807351 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.807328    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j\" (UID: \"0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j"
Nov 03 00:08:51.807423 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.807398    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j\" (UID: \"0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j"
Nov 03 00:08:51.833387 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.833215    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jpbw\" (UniqueName: \"kubernetes.io/projected/0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945-kube-api-access-6jpbw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j\" (UID: \"0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j"
Nov 03 00:08:51.867257 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.867194    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j"
Nov 03 00:08:51.884610 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.884585    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2"]
Nov 03 00:08:51.886346 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:08:51.886318    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e7a1a7_b210_41d5_af65_4722b7b33e67.slice/crio-df7f195f0218436598ca29caf65711f6ce78dd610141b9be92e73dcd61e9bfdf WatchSource:0}: Error finding container df7f195f0218436598ca29caf65711f6ce78dd610141b9be92e73dcd61e9bfdf: Status 404 returned error can't find the container with id df7f195f0218436598ca29caf65711f6ce78dd610141b9be92e73dcd61e9bfdf
Nov 03 00:08:51.972365 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:51.972333    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9"]
Nov 03 00:08:51.976077 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:08:51.976052    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b7bc3d_86d7_4dc5_93d5_8f213d700093.slice/crio-4b6383d259e1b59103bdb14c0dbaaa0d3e1a00f0ea9aad2bc91ac98abcd4b6bb WatchSource:0}: Error finding container 4b6383d259e1b59103bdb14c0dbaaa0d3e1a00f0ea9aad2bc91ac98abcd4b6bb: Status 404 returned error can't find the container with id 4b6383d259e1b59103bdb14c0dbaaa0d3e1a00f0ea9aad2bc91ac98abcd4b6bb
Nov 03 00:08:52.075289 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:52.075260    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c"]
Nov 03 00:08:52.083441 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:08:52.079691    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda276be2c_84f0_4457_91fa_d5dedacaa865.slice/crio-a64c63819a456fbb86b110d2fce1f49be15af26be9d32745a3d11febb4cd9e1a WatchSource:0}: Error finding container a64c63819a456fbb86b110d2fce1f49be15af26be9d32745a3d11febb4cd9e1a: Status 404 returned error can't find the container with id a64c63819a456fbb86b110d2fce1f49be15af26be9d32745a3d11febb4cd9e1a
Nov 03 00:08:52.099467 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:52.099441    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg"]
Nov 03 00:08:52.103633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:52.103613    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg"
Nov 03 00:08:52.118587 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:52.118509    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-sqshm\""
Nov 03 00:08:52.119233 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:52.119210    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg"]
Nov 03 00:08:52.170257 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:52.170229    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j"]
Nov 03 00:08:52.191662 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:52.191632    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9" event={"ID":"15b7bc3d-86d7-4dc5-93d5-8f213d700093","Type":"ContainerStarted","Data":"4b6383d259e1b59103bdb14c0dbaaa0d3e1a00f0ea9aad2bc91ac98abcd4b6bb"}
Nov 03 00:08:52.193626 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:52.193596    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c" event={"ID":"a276be2c-84f0-4457-91fa-d5dedacaa865","Type":"ContainerStarted","Data":"a64c63819a456fbb86b110d2fce1f49be15af26be9d32745a3d11febb4cd9e1a"}
Nov 03 00:08:52.195365 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:52.195298    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md" event={"ID":"acb40932-dcac-4068-a88b-1479f7904f3c","Type":"ContainerStarted","Data":"f9dae51b9f74192925ac514132dcb4dcf777ee74af929013ca57c2707fcb1b7d"}
Nov 03 00:08:52.197484 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:52.197456    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j" event={"ID":"0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945","Type":"ContainerStarted","Data":"8399f7509e9def53946c3f40d9139259518f01e583ccc797dccfed5b132bb918"}
Nov 03 00:08:52.199438 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:52.199412    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2" event={"ID":"96e7a1a7-b210-41d5-af65-4722b7b33e67","Type":"ContainerStarted","Data":"df7f195f0218436598ca29caf65711f6ce78dd610141b9be92e73dcd61e9bfdf"}
Nov 03 00:08:52.209839 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:52.209813    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07a769f1-c30f-4832-8159-fd6108718ffd-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg\" (UID: \"07a769f1-c30f-4832-8159-fd6108718ffd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg"
Nov 03 00:08:52.209954 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:52.209929    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dvdm\" (UniqueName: \"kubernetes.io/projected/07a769f1-c30f-4832-8159-fd6108718ffd-kube-api-access-5dvdm\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg\" (UID: \"07a769f1-c30f-4832-8159-fd6108718ffd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg"
Nov 03 00:08:52.210025 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:52.209991    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07a769f1-c30f-4832-8159-fd6108718ffd-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg\" (UID: \"07a769f1-c30f-4832-8159-fd6108718ffd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg"
Nov 03 00:08:52.311474 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:52.310376    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07a769f1-c30f-4832-8159-fd6108718ffd-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg\" (UID: \"07a769f1-c30f-4832-8159-fd6108718ffd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg"
Nov 03 00:08:52.311474 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:52.310427    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07a769f1-c30f-4832-8159-fd6108718ffd-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg\" (UID: \"07a769f1-c30f-4832-8159-fd6108718ffd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg"
Nov 03 00:08:52.311474 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:52.310527    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dvdm\" (UniqueName: \"kubernetes.io/projected/07a769f1-c30f-4832-8159-fd6108718ffd-kube-api-access-5dvdm\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg\" (UID: \"07a769f1-c30f-4832-8159-fd6108718ffd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg"
Nov 03 00:08:52.311474 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:52.311182    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07a769f1-c30f-4832-8159-fd6108718ffd-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg\" (UID: \"07a769f1-c30f-4832-8159-fd6108718ffd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg"
Nov 03 00:08:52.311474 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:52.311429    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07a769f1-c30f-4832-8159-fd6108718ffd-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg\" (UID: \"07a769f1-c30f-4832-8159-fd6108718ffd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg"
Nov 03 00:08:52.337970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:52.328228    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dvdm\" (UniqueName: \"kubernetes.io/projected/07a769f1-c30f-4832-8159-fd6108718ffd-kube-api-access-5dvdm\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg\" (UID: \"07a769f1-c30f-4832-8159-fd6108718ffd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg"
Nov 03 00:08:52.424433 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:52.424159    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg"
Nov 03 00:08:52.773410 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:52.773301    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg"]
Nov 03 00:08:52.777129 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:08:52.777097    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a769f1_c30f_4832_8159_fd6108718ffd.slice/crio-94e95eb61bdf261c3fc9728e9f9803fd83fe36cf219eff1de15ecee7c4335bc5 WatchSource:0}: Error finding container 94e95eb61bdf261c3fc9728e9f9803fd83fe36cf219eff1de15ecee7c4335bc5: Status 404 returned error can't find the container with id 94e95eb61bdf261c3fc9728e9f9803fd83fe36cf219eff1de15ecee7c4335bc5
Nov 03 00:08:53.227250 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:53.227213    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg" event={"ID":"07a769f1-c30f-4832-8159-fd6108718ffd","Type":"ContainerStarted","Data":"94e95eb61bdf261c3fc9728e9f9803fd83fe36cf219eff1de15ecee7c4335bc5"}
Nov 03 00:08:55.243541 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:55.243501    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms"]
Nov 03 00:08:55.248508 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:55.248436    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms"
Nov 03 00:08:55.261202 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:55.261057    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms"]
Nov 03 00:08:55.342411 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:55.342108    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c79292b-876b-4574-9238-2f29d04926fb-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms\" (UID: \"9c79292b-876b-4574-9238-2f29d04926fb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms"
Nov 03 00:08:55.342411 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:55.342173    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r922g\" (UniqueName: \"kubernetes.io/projected/9c79292b-876b-4574-9238-2f29d04926fb-kube-api-access-r922g\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms\" (UID: \"9c79292b-876b-4574-9238-2f29d04926fb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms"
Nov 03 00:08:55.342411 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:55.342234    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c79292b-876b-4574-9238-2f29d04926fb-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms\" (UID: \"9c79292b-876b-4574-9238-2f29d04926fb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms"
Nov 03 00:08:55.444781 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:55.443581    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c79292b-876b-4574-9238-2f29d04926fb-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms\" (UID: \"9c79292b-876b-4574-9238-2f29d04926fb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms"
Nov 03 00:08:55.444781 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:55.443639    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r922g\" (UniqueName: \"kubernetes.io/projected/9c79292b-876b-4574-9238-2f29d04926fb-kube-api-access-r922g\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms\" (UID: \"9c79292b-876b-4574-9238-2f29d04926fb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms"
Nov 03 00:08:55.444781 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:55.443691    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c79292b-876b-4574-9238-2f29d04926fb-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms\" (UID: \"9c79292b-876b-4574-9238-2f29d04926fb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms"
Nov 03 00:08:55.444781 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:55.444165    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c79292b-876b-4574-9238-2f29d04926fb-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms\" (UID: \"9c79292b-876b-4574-9238-2f29d04926fb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms"
Nov 03 00:08:55.444781 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:55.444425    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c79292b-876b-4574-9238-2f29d04926fb-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms\" (UID: \"9c79292b-876b-4574-9238-2f29d04926fb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms"
Nov 03 00:08:55.474586 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:55.474429    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r922g\" (UniqueName: \"kubernetes.io/projected/9c79292b-876b-4574-9238-2f29d04926fb-kube-api-access-r922g\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms\" (UID: \"9c79292b-876b-4574-9238-2f29d04926fb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms"
Nov 03 00:08:55.581174 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:55.581102    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms"
Nov 03 00:08:55.953392 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:55.953360    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms"]
Nov 03 00:08:55.967056 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:08:55.967026    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c79292b_876b_4574_9238_2f29d04926fb.slice/crio-0439b15a3f7855638b0842367e2775436a674718193378724e08a3d9290a160a WatchSource:0}: Error finding container 0439b15a3f7855638b0842367e2775436a674718193378724e08a3d9290a160a: Status 404 returned error can't find the container with id 0439b15a3f7855638b0842367e2775436a674718193378724e08a3d9290a160a
Nov 03 00:08:56.264847 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.264761    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms" event={"ID":"9c79292b-876b-4574-9238-2f29d04926fb","Type":"ContainerStarted","Data":"0439b15a3f7855638b0842367e2775436a674718193378724e08a3d9290a160a"}
Nov 03 00:08:56.364619 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.364581    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"]
Nov 03 00:08:56.369076 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.368595    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.386214 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.386138    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v9b8n\""
Nov 03 00:08:56.390639 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.390598    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 00:08:56.404153 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.404086    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"]
Nov 03 00:08:56.458291 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.457766    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.458291 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.457816    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.458291 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.457845    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv2h7\" (UniqueName: \"kubernetes.io/projected/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-kube-api-access-vv2h7\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.458291 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.457877    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.458291 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.457922    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.458291 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.457955    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.458291 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.457987    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.458291 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.458018    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.458291 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.458102    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.492382 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.492349    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"]
Nov 03 00:08:56.497041 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.496332    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.516380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.516299    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-7cq6p\""
Nov 03 00:08:56.519433 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.519394    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"]
Nov 03 00:08:56.560555 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.559363    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.560555 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.559425    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.560555 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.559460    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.560555 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.559484    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv2h7\" (UniqueName: \"kubernetes.io/projected/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-kube-api-access-vv2h7\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.560555 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.559517    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.560555 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.559547    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.560555 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.559580    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.560555 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.559609    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.560555 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.559640    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.560555 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.560099    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.562354 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.562001    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.562354 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.562282    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.562513 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.562393    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.562513 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.562454    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.563791 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.563767    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.568409 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.568374    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.584733 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.584707    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.591234 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.591207    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv2h7\" (UniqueName: \"kubernetes.io/projected/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-kube-api-access-vv2h7\") pod \"maas-default-gateway-openshift-default-6cc458d887-8fzrc\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.661363 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.661325    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.661521 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.661386    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.661521 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.661436    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.661521 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.661470    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.661521 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.661495    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kbrj\" (UniqueName: \"kubernetes.io/projected/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-kube-api-access-7kbrj\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.661736 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.661593    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.661736 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.661626    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.661736 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.661651    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.661736 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.661688    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.688115 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.687811    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:08:56.762941 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.762631    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.762941 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.762690    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.762941 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.762741    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.762941 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.762776    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.762941 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.762802    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kbrj\" (UniqueName: \"kubernetes.io/projected/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-kube-api-access-7kbrj\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.762941 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.762848    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.762941 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.762872    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.762941 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.762886    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.762941 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.762928    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.764106 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.764076    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.764364 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.764343    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.764574 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.764555    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.764762 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.764744    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.765225 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.765202    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.766100 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.766076    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.777676 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.777601    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.794124 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.794077    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kbrj\" (UniqueName: \"kubernetes.io/projected/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-kube-api-access-7kbrj\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.808495 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.808401    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6\" (UID: \"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:56.816360 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:56.816116    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:08:57.031371 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.031291    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55"]
Nov 03 00:08:57.036101 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.036064    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55"
Nov 03 00:08:57.055564 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.055539    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55"]
Nov 03 00:08:57.061443 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.060884    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt"]
Nov 03 00:08:57.070226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.070203    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt"
Nov 03 00:08:57.093085 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.092769    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt"]
Nov 03 00:08:57.098419 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.096542    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6"]
Nov 03 00:08:57.101982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.101018    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6"
Nov 03 00:08:57.101982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.101930    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"]
Nov 03 00:08:57.135192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.135161    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6"]
Nov 03 00:08:57.153589 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.153557    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv"]
Nov 03 00:08:57.160850 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.160817    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv"]
Nov 03 00:08:57.161182 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.161165    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv"
Nov 03 00:08:57.170541 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.170374    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49ec319e-e958-40d8-89de-c58335aeeea0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt\" (UID: \"49ec319e-e958-40d8-89de-c58335aeeea0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt"
Nov 03 00:08:57.170541 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.170441    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxrmq\" (UniqueName: \"kubernetes.io/projected/49ec319e-e958-40d8-89de-c58335aeeea0-kube-api-access-rxrmq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt\" (UID: \"49ec319e-e958-40d8-89de-c58335aeeea0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt"
Nov 03 00:08:57.170541 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.170526    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/251d8990-c450-4333-8dca-456dd9640e09-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55\" (UID: \"251d8990-c450-4333-8dca-456dd9640e09\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55"
Nov 03 00:08:57.170759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.170563    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/251d8990-c450-4333-8dca-456dd9640e09-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55\" (UID: \"251d8990-c450-4333-8dca-456dd9640e09\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55"
Nov 03 00:08:57.170759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.170587    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9tjx\" (UniqueName: \"kubernetes.io/projected/251d8990-c450-4333-8dca-456dd9640e09-kube-api-access-q9tjx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55\" (UID: \"251d8990-c450-4333-8dca-456dd9640e09\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55"
Nov 03 00:08:57.170759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.170643    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49ec319e-e958-40d8-89de-c58335aeeea0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt\" (UID: \"49ec319e-e958-40d8-89de-c58335aeeea0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt"
Nov 03 00:08:57.213711 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.213679    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"]
Nov 03 00:08:57.220208 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:08:57.220179    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9b7722_1b39_4fd2_ba12_aa6668d1c8dc.slice/crio-20291b7c043b2dc731c750019638e638a23fce184f85e3664abf562f6c97ff04 WatchSource:0}: Error finding container 20291b7c043b2dc731c750019638e638a23fce184f85e3664abf562f6c97ff04: Status 404 returned error can't find the container with id 20291b7c043b2dc731c750019638e638a23fce184f85e3664abf562f6c97ff04
Nov 03 00:08:57.272076 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.272013    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrgd4\" (UniqueName: \"kubernetes.io/projected/997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca-kube-api-access-hrgd4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv\" (UID: \"997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv"
Nov 03 00:08:57.272076 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.272059    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49ec319e-e958-40d8-89de-c58335aeeea0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt\" (UID: \"49ec319e-e958-40d8-89de-c58335aeeea0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt"
Nov 03 00:08:57.272564 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.272105    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxrmq\" (UniqueName: \"kubernetes.io/projected/49ec319e-e958-40d8-89de-c58335aeeea0-kube-api-access-rxrmq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt\" (UID: \"49ec319e-e958-40d8-89de-c58335aeeea0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt"
Nov 03 00:08:57.272564 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.272141    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47d9572f-cce3-40b5-a8e0-7a9fb348e127-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6\" (UID: \"47d9572f-cce3-40b5-a8e0-7a9fb348e127\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6"
Nov 03 00:08:57.272564 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.272178    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv\" (UID: \"997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv"
Nov 03 00:08:57.272564 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.272236    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv\" (UID: \"997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv"
Nov 03 00:08:57.272564 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.272258    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47d9572f-cce3-40b5-a8e0-7a9fb348e127-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6\" (UID: \"47d9572f-cce3-40b5-a8e0-7a9fb348e127\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6"
Nov 03 00:08:57.272564 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.272286    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xkgw\" (UniqueName: \"kubernetes.io/projected/47d9572f-cce3-40b5-a8e0-7a9fb348e127-kube-api-access-6xkgw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6\" (UID: \"47d9572f-cce3-40b5-a8e0-7a9fb348e127\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6"
Nov 03 00:08:57.272564 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.272313    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/251d8990-c450-4333-8dca-456dd9640e09-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55\" (UID: \"251d8990-c450-4333-8dca-456dd9640e09\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55"
Nov 03 00:08:57.272564 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.272347    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/251d8990-c450-4333-8dca-456dd9640e09-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55\" (UID: \"251d8990-c450-4333-8dca-456dd9640e09\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55"
Nov 03 00:08:57.272564 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.272373    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9tjx\" (UniqueName: \"kubernetes.io/projected/251d8990-c450-4333-8dca-456dd9640e09-kube-api-access-q9tjx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55\" (UID: \"251d8990-c450-4333-8dca-456dd9640e09\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55"
Nov 03 00:08:57.272564 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.272432    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49ec319e-e958-40d8-89de-c58335aeeea0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt\" (UID: \"49ec319e-e958-40d8-89de-c58335aeeea0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt"
Nov 03 00:08:57.273090 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.272844    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49ec319e-e958-40d8-89de-c58335aeeea0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt\" (UID: \"49ec319e-e958-40d8-89de-c58335aeeea0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt"
Nov 03 00:08:57.275281 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.274715    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/251d8990-c450-4333-8dca-456dd9640e09-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55\" (UID: \"251d8990-c450-4333-8dca-456dd9640e09\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55"
Nov 03 00:08:57.275281 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.275012    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/251d8990-c450-4333-8dca-456dd9640e09-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55\" (UID: \"251d8990-c450-4333-8dca-456dd9640e09\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55"
Nov 03 00:08:57.275788 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.275737    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49ec319e-e958-40d8-89de-c58335aeeea0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt\" (UID: \"49ec319e-e958-40d8-89de-c58335aeeea0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt"
Nov 03 00:08:57.294660 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.294555    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6" event={"ID":"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc","Type":"ContainerStarted","Data":"20291b7c043b2dc731c750019638e638a23fce184f85e3664abf562f6c97ff04"}
Nov 03 00:08:57.300704 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.300653    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9tjx\" (UniqueName: \"kubernetes.io/projected/251d8990-c450-4333-8dca-456dd9640e09-kube-api-access-q9tjx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55\" (UID: \"251d8990-c450-4333-8dca-456dd9640e09\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55"
Nov 03 00:08:57.301650 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.301600    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxrmq\" (UniqueName: \"kubernetes.io/projected/49ec319e-e958-40d8-89de-c58335aeeea0-kube-api-access-rxrmq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt\" (UID: \"49ec319e-e958-40d8-89de-c58335aeeea0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt"
Nov 03 00:08:57.315950 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.315868    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc" event={"ID":"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c","Type":"ContainerStarted","Data":"21fff050a27d4a3646ad0e42b34501d514ae757ccf9322b53a1457bb1bfb12d6"}
Nov 03 00:08:57.370095 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.369823    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55"
Nov 03 00:08:57.373118 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.373077    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47d9572f-cce3-40b5-a8e0-7a9fb348e127-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6\" (UID: \"47d9572f-cce3-40b5-a8e0-7a9fb348e127\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6"
Nov 03 00:08:57.373248 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.373128    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv\" (UID: \"997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv"
Nov 03 00:08:57.373248 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.373193    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv\" (UID: \"997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv"
Nov 03 00:08:57.373362 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.373270    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47d9572f-cce3-40b5-a8e0-7a9fb348e127-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6\" (UID: \"47d9572f-cce3-40b5-a8e0-7a9fb348e127\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6"
Nov 03 00:08:57.373362 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.373297    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xkgw\" (UniqueName: \"kubernetes.io/projected/47d9572f-cce3-40b5-a8e0-7a9fb348e127-kube-api-access-6xkgw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6\" (UID: \"47d9572f-cce3-40b5-a8e0-7a9fb348e127\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6"
Nov 03 00:08:57.373468 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.373387    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrgd4\" (UniqueName: \"kubernetes.io/projected/997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca-kube-api-access-hrgd4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv\" (UID: \"997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv"
Nov 03 00:08:57.374821 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.374195    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47d9572f-cce3-40b5-a8e0-7a9fb348e127-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6\" (UID: \"47d9572f-cce3-40b5-a8e0-7a9fb348e127\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6"
Nov 03 00:08:57.374821 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.374319    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv\" (UID: \"997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv"
Nov 03 00:08:57.374821 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.374537    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47d9572f-cce3-40b5-a8e0-7a9fb348e127-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6\" (UID: \"47d9572f-cce3-40b5-a8e0-7a9fb348e127\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6"
Nov 03 00:08:57.374821 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.374588    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv\" (UID: \"997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv"
Nov 03 00:08:57.397840 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.397773    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrgd4\" (UniqueName: \"kubernetes.io/projected/997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca-kube-api-access-hrgd4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv\" (UID: \"997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv"
Nov 03 00:08:57.400641 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.400434    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xkgw\" (UniqueName: \"kubernetes.io/projected/47d9572f-cce3-40b5-a8e0-7a9fb348e127-kube-api-access-6xkgw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6\" (UID: \"47d9572f-cce3-40b5-a8e0-7a9fb348e127\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6"
Nov 03 00:08:57.403862 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.403668    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt"
Nov 03 00:08:57.481204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.479218    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6"
Nov 03 00:08:57.485876 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.485851    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv"
Nov 03 00:08:57.730137 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.730057    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55"]
Nov 03 00:08:57.740633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.740499    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt"]
Nov 03 00:08:57.813854 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.813822    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6"]
Nov 03 00:08:57.821504 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:08:57.821476    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d9572f_cce3_40b5_a8e0_7a9fb348e127.slice/crio-3164f1e22a2c6b44902b53d252c9fc86ca07e2a6f5c2bb163494f75320dedd66 WatchSource:0}: Error finding container 3164f1e22a2c6b44902b53d252c9fc86ca07e2a6f5c2bb163494f75320dedd66: Status 404 returned error can't find the container with id 3164f1e22a2c6b44902b53d252c9fc86ca07e2a6f5c2bb163494f75320dedd66
Nov 03 00:08:57.885680 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:57.885656    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv"]
Nov 03 00:08:57.889226 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:08:57.889198    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997c7e6a_b7f7_4dcb_8e72_a0e6c90c34ca.slice/crio-c37026178c095545b0ad4e707990e2f335cce81b3cfe69e93dfcf2d5e86ffffe WatchSource:0}: Error finding container c37026178c095545b0ad4e707990e2f335cce81b3cfe69e93dfcf2d5e86ffffe: Status 404 returned error can't find the container with id c37026178c095545b0ad4e707990e2f335cce81b3cfe69e93dfcf2d5e86ffffe
Nov 03 00:08:58.361596 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:58.361500    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6" event={"ID":"47d9572f-cce3-40b5-a8e0-7a9fb348e127","Type":"ContainerStarted","Data":"3164f1e22a2c6b44902b53d252c9fc86ca07e2a6f5c2bb163494f75320dedd66"}
Nov 03 00:08:58.393926 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:58.393878    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv" event={"ID":"997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca","Type":"ContainerStarted","Data":"c37026178c095545b0ad4e707990e2f335cce81b3cfe69e93dfcf2d5e86ffffe"}
Nov 03 00:08:58.423926 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:58.423419    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55" event={"ID":"251d8990-c450-4333-8dca-456dd9640e09","Type":"ContainerStarted","Data":"00934c965ece9bf846ca745096e799e8cc4c68bfa030f83cd96b5d4b099c0795"}
Nov 03 00:08:58.458311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:08:58.458271    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt" event={"ID":"49ec319e-e958-40d8-89de-c58335aeeea0","Type":"ContainerStarted","Data":"ed178dfd57575d89b0fab846152d5a7a215da75e652ccce97138022c423c4386"}
Nov 03 00:09:07.419333 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:09:07.419240    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:09:09.313696 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:09.313660    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:09:09.314111 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:09.313747    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:09:09.314111 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:09.313793    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:09:09.461781 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:09.461743    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:09:09.461889 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:09.461837    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:09:09.462024 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:09.461886    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:09:09.667231 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:09.666904    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6" event={"ID":"3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc","Type":"ContainerStarted","Data":"72e4763df358314c2a948918584f17548531255988868c3bec3208a1c4561a8e"}
Nov 03 00:09:09.823028 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:09.816661    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:09:10.697275 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:10.696960    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc" event={"ID":"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c","Type":"ContainerStarted","Data":"18a8893cac47b63fca1c159aea7507e8c7a080459cbadd0005de6e25326804d6"}
Nov 03 00:09:10.757852 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:10.756465    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6" podStartSLOduration=2.668168505 podStartE2EDuration="14.756443409s" totalImagesPullingTime="12.088274904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:08:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:09.727018076 +0000 UTC m=+232.606602120" watchObservedRunningTime="2026-11-03 00:09:10.756443409 +0000 UTC m=+233.636027453"
Nov 03 00:09:10.757852 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:10.757205    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc" podStartSLOduration=2.410539911 podStartE2EDuration="14.757194375s" totalImagesPullingTime="12.346654464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:08:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:10.746844136 +0000 UTC m=+233.626428180" watchObservedRunningTime="2026-11-03 00:09:10.757194375 +0000 UTC m=+233.636778419"
Nov 03 00:09:10.823732 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:10.823452    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:09:11.688607 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:11.688578    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:09:11.688746 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:11.688701    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:09:11.695047 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:11.695025    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:09:11.703165 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:11.703148    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:09:11.706063 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:11.705126    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6"
Nov 03 00:09:11.706063 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:11.705857    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:09:12.714399 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:12.713506    5012 generic.go:356] "Generic (PLEG): container finished" podID="0158fe4a-e132-4d0e-92e9-8ce181c2c559" containerID="2d07cd551f649183653b0539db2c7f8ad434bc18ea065b39e4c887ee5a711f61" exitCode=0
Nov 03 00:09:12.714399 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:12.713628    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b" event={"ID":"0158fe4a-e132-4d0e-92e9-8ce181c2c559","Type":"ContainerDied","Data":"2d07cd551f649183653b0539db2c7f8ad434bc18ea065b39e4c887ee5a711f61"}
Nov 03 00:09:13.726262 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:13.726215    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md" event={"ID":"acb40932-dcac-4068-a88b-1479f7904f3c","Type":"ContainerStarted","Data":"e89575d3684c2d14c6817b4045f9b73e56077b0520dc4f89bf4c7d4682ad8a3d"}
Nov 03 00:09:13.729716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:13.729681    5012 generic.go:356] "Generic (PLEG): container finished" podID="47d9572f-cce3-40b5-a8e0-7a9fb348e127" containerID="28a37f7bb7802cc4c9784db2d20b7e92582bf0730031eb55ef19abe0fa8c009f" exitCode=0
Nov 03 00:09:13.730805 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:13.730171    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6" event={"ID":"47d9572f-cce3-40b5-a8e0-7a9fb348e127","Type":"ContainerDied","Data":"28a37f7bb7802cc4c9784db2d20b7e92582bf0730031eb55ef19abe0fa8c009f"}
Nov 03 00:09:13.734036 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:13.733774    5012 generic.go:356] "Generic (PLEG): container finished" podID="997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca" containerID="62d54ee640c79a202447ddbadfabe811560a8dc6e5102cacca540d1f54c56ca1" exitCode=0
Nov 03 00:09:13.734886 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:13.734862    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv" event={"ID":"997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca","Type":"ContainerDied","Data":"62d54ee640c79a202447ddbadfabe811560a8dc6e5102cacca540d1f54c56ca1"}
Nov 03 00:09:14.740142 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:14.740105    5012 generic.go:356] "Generic (PLEG): container finished" podID="49ec319e-e958-40d8-89de-c58335aeeea0" containerID="7f307810097bc997b62967bee0ec68567f6918492a12a03dc6b5d7cc8c9d960b" exitCode=0
Nov 03 00:09:14.740540 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:14.740191    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt" event={"ID":"49ec319e-e958-40d8-89de-c58335aeeea0","Type":"ContainerDied","Data":"7f307810097bc997b62967bee0ec68567f6918492a12a03dc6b5d7cc8c9d960b"}
Nov 03 00:09:14.741956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:14.741932    5012 generic.go:356] "Generic (PLEG): container finished" podID="a276be2c-84f0-4457-91fa-d5dedacaa865" containerID="803606877bcaf9fc4ce1e008b782fa5059c28bd27da8a9702930aa232c4f2bba" exitCode=0
Nov 03 00:09:14.742066 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:14.742003    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c" event={"ID":"a276be2c-84f0-4457-91fa-d5dedacaa865","Type":"ContainerDied","Data":"803606877bcaf9fc4ce1e008b782fa5059c28bd27da8a9702930aa232c4f2bba"}
Nov 03 00:09:14.743875 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:14.743849    5012 generic.go:356] "Generic (PLEG): container finished" podID="acb40932-dcac-4068-a88b-1479f7904f3c" containerID="e89575d3684c2d14c6817b4045f9b73e56077b0520dc4f89bf4c7d4682ad8a3d" exitCode=0
Nov 03 00:09:14.743991 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:14.743939    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md" event={"ID":"acb40932-dcac-4068-a88b-1479f7904f3c","Type":"ContainerDied","Data":"e89575d3684c2d14c6817b4045f9b73e56077b0520dc4f89bf4c7d4682ad8a3d"}
Nov 03 00:09:14.745838 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:14.745817    5012 generic.go:356] "Generic (PLEG): container finished" podID="251d8990-c450-4333-8dca-456dd9640e09" containerID="7c38d88a4450f43efc2f0baab68c79ef6f20c29bddf59a21fa27466d4996e5c8" exitCode=0
Nov 03 00:09:14.746078 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:14.745899    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55" event={"ID":"251d8990-c450-4333-8dca-456dd9640e09","Type":"ContainerDied","Data":"7c38d88a4450f43efc2f0baab68c79ef6f20c29bddf59a21fa27466d4996e5c8"}
Nov 03 00:09:15.752174 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:15.752143    5012 generic.go:356] "Generic (PLEG): container finished" podID="15b7bc3d-86d7-4dc5-93d5-8f213d700093" containerID="c4707efa01413611a4667e2a02b0b22a5189227b10dba82f6a0d84a7eceab5c8" exitCode=0
Nov 03 00:09:15.752517 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:15.752221    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9" event={"ID":"15b7bc3d-86d7-4dc5-93d5-8f213d700093","Type":"ContainerDied","Data":"c4707efa01413611a4667e2a02b0b22a5189227b10dba82f6a0d84a7eceab5c8"}
Nov 03 00:09:15.755933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:15.755494    5012 generic.go:356] "Generic (PLEG): container finished" podID="a276be2c-84f0-4457-91fa-d5dedacaa865" containerID="569638e72e7635b936681c761691b550926f4d6b3f09da1bd77593844c2b0bf0" exitCode=0
Nov 03 00:09:15.755933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:15.755623    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c" event={"ID":"a276be2c-84f0-4457-91fa-d5dedacaa865","Type":"ContainerDied","Data":"569638e72e7635b936681c761691b550926f4d6b3f09da1bd77593844c2b0bf0"}
Nov 03 00:09:15.757644 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:15.757615    5012 generic.go:356] "Generic (PLEG): container finished" podID="0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945" containerID="c5b4f5c3949b01e89edee2af6591cde1a869c82d433305963b98211e3a5bba22" exitCode=0
Nov 03 00:09:15.757731 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:15.757647    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j" event={"ID":"0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945","Type":"ContainerDied","Data":"c5b4f5c3949b01e89edee2af6591cde1a869c82d433305963b98211e3a5bba22"}
Nov 03 00:09:15.759527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:15.759436    5012 generic.go:356] "Generic (PLEG): container finished" podID="96e7a1a7-b210-41d5-af65-4722b7b33e67" containerID="f4ec2c4d435f4a6c3b4e4687b6c33badfd096ef2cdae7a72ddaaf1169dbb9edd" exitCode=0
Nov 03 00:09:15.759527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:15.759504    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2" event={"ID":"96e7a1a7-b210-41d5-af65-4722b7b33e67","Type":"ContainerDied","Data":"f4ec2c4d435f4a6c3b4e4687b6c33badfd096ef2cdae7a72ddaaf1169dbb9edd"}
Nov 03 00:09:15.761963 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:15.761937    5012 generic.go:356] "Generic (PLEG): container finished" podID="07a769f1-c30f-4832-8159-fd6108718ffd" containerID="938f5fb68d1956bae1de36fac58c960d9fad288215bf20418753356544a17b81" exitCode=0
Nov 03 00:09:15.762290 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:15.762238    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg" event={"ID":"07a769f1-c30f-4832-8159-fd6108718ffd","Type":"ContainerDied","Data":"938f5fb68d1956bae1de36fac58c960d9fad288215bf20418753356544a17b81"}
Nov 03 00:09:15.766403 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:15.764510    5012 generic.go:356] "Generic (PLEG): container finished" podID="9c79292b-876b-4574-9238-2f29d04926fb" containerID="de484b77b3778144fff7b6a24997d35b0563821be6e9a2bb6c736cbca99700b8" exitCode=0
Nov 03 00:09:15.766403 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:15.764546    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms" event={"ID":"9c79292b-876b-4574-9238-2f29d04926fb","Type":"ContainerDied","Data":"de484b77b3778144fff7b6a24997d35b0563821be6e9a2bb6c736cbca99700b8"}
Nov 03 00:09:16.779109 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:16.778448    5012 generic.go:356] "Generic (PLEG): container finished" podID="47d9572f-cce3-40b5-a8e0-7a9fb348e127" containerID="c2068f6bdd213fa0b30587df31dee871c9cd24c015f22f6f1010fbd2f9411dd7" exitCode=0
Nov 03 00:09:16.779109 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:16.778580    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6" event={"ID":"47d9572f-cce3-40b5-a8e0-7a9fb348e127","Type":"ContainerDied","Data":"c2068f6bdd213fa0b30587df31dee871c9cd24c015f22f6f1010fbd2f9411dd7"}
Nov 03 00:09:16.782838 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:16.782810    5012 generic.go:356] "Generic (PLEG): container finished" podID="0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945" containerID="e52187694cd89c69648e5448bc0b95a67da649e2f338b5e02ae7486cdf096116" exitCode=0
Nov 03 00:09:16.782934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:16.782893    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j" event={"ID":"0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945","Type":"ContainerDied","Data":"e52187694cd89c69648e5448bc0b95a67da649e2f338b5e02ae7486cdf096116"}
Nov 03 00:09:16.786934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:16.786752    5012 generic.go:356] "Generic (PLEG): container finished" podID="96e7a1a7-b210-41d5-af65-4722b7b33e67" containerID="760f77ceb7849186b7880fe20bd5e9fdd51534a9d5d64e5aae70257eda92439a" exitCode=0
Nov 03 00:09:16.786934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:16.786853    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2" event={"ID":"96e7a1a7-b210-41d5-af65-4722b7b33e67","Type":"ContainerDied","Data":"760f77ceb7849186b7880fe20bd5e9fdd51534a9d5d64e5aae70257eda92439a"}
Nov 03 00:09:16.792087 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:16.791956    5012 generic.go:356] "Generic (PLEG): container finished" podID="997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca" containerID="6925a20be57075faedcf5494e73ae11da12531c3edc65e674457017ebeff7bdf" exitCode=0
Nov 03 00:09:16.792087 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:16.792033    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv" event={"ID":"997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca","Type":"ContainerDied","Data":"6925a20be57075faedcf5494e73ae11da12531c3edc65e674457017ebeff7bdf"}
Nov 03 00:09:16.795482 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:16.794724    5012 generic.go:356] "Generic (PLEG): container finished" podID="251d8990-c450-4333-8dca-456dd9640e09" containerID="fafd40377324db7a53809fa7d1e8963cb6450f75a632d8a9e37f137073ecb31b" exitCode=0
Nov 03 00:09:16.795482 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:16.794878    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55" event={"ID":"251d8990-c450-4333-8dca-456dd9640e09","Type":"ContainerDied","Data":"fafd40377324db7a53809fa7d1e8963cb6450f75a632d8a9e37f137073ecb31b"}
Nov 03 00:09:16.799733 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:16.799148    5012 generic.go:356] "Generic (PLEG): container finished" podID="15b7bc3d-86d7-4dc5-93d5-8f213d700093" containerID="1f667208ce4c75ebb82e380622acb948acc768dffe55ccf12ccf78134abea3b1" exitCode=0
Nov 03 00:09:16.799733 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:16.799214    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9" event={"ID":"15b7bc3d-86d7-4dc5-93d5-8f213d700093","Type":"ContainerDied","Data":"1f667208ce4c75ebb82e380622acb948acc768dffe55ccf12ccf78134abea3b1"}
Nov 03 00:09:16.803943 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:16.803843    5012 generic.go:356] "Generic (PLEG): container finished" podID="49ec319e-e958-40d8-89de-c58335aeeea0" containerID="a9a2c646e47869b4f3140b89d40c74dafadf7374e1a5e76eb5ccff5ae7088fb5" exitCode=0
Nov 03 00:09:16.804651 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:16.804015    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt" event={"ID":"49ec319e-e958-40d8-89de-c58335aeeea0","Type":"ContainerDied","Data":"a9a2c646e47869b4f3140b89d40c74dafadf7374e1a5e76eb5ccff5ae7088fb5"}
Nov 03 00:09:16.820057 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:16.820033    5012 generic.go:356] "Generic (PLEG): container finished" podID="acb40932-dcac-4068-a88b-1479f7904f3c" containerID="cf8d094b3a2ebaed47bba17302abc0170ef2718cee742d86691ea1e752fb41ec" exitCode=0
Nov 03 00:09:16.820202 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:16.820185    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md" event={"ID":"acb40932-dcac-4068-a88b-1479f7904f3c","Type":"ContainerDied","Data":"cf8d094b3a2ebaed47bba17302abc0170ef2718cee742d86691ea1e752fb41ec"}
Nov 03 00:09:19.861402 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:19.861362    5012 generic.go:356] "Generic (PLEG): container finished" podID="0158fe4a-e132-4d0e-92e9-8ce181c2c559" containerID="7afb331241e0d2beda93cbb5eee2dec5510635ccd4f46d781781586239e5b220" exitCode=0
Nov 03 00:09:19.861888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:19.861464    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b" event={"ID":"0158fe4a-e132-4d0e-92e9-8ce181c2c559","Type":"ContainerDied","Data":"7afb331241e0d2beda93cbb5eee2dec5510635ccd4f46d781781586239e5b220"}
Nov 03 00:09:22.927942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:22.927693    5012 generic.go:356] "Generic (PLEG): container finished" podID="9c79292b-876b-4574-9238-2f29d04926fb" containerID="17ad480b1a319a46e267d79cab8b8fffe0bec9ff1decf35d06b9b73c6595fb5e" exitCode=0
Nov 03 00:09:22.927942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:22.927753    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms" event={"ID":"9c79292b-876b-4574-9238-2f29d04926fb","Type":"ContainerDied","Data":"17ad480b1a319a46e267d79cab8b8fffe0bec9ff1decf35d06b9b73c6595fb5e"}
Nov 03 00:09:23.950602 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:23.950559    5012 generic.go:356] "Generic (PLEG): container finished" podID="07a769f1-c30f-4832-8159-fd6108718ffd" containerID="48a56e13bcca50b1d7852ae484159a3909061951e8c59aaadbb4b071e02dd02d" exitCode=0
Nov 03 00:09:23.951268 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:23.950622    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg" event={"ID":"07a769f1-c30f-4832-8159-fd6108718ffd","Type":"ContainerDied","Data":"48a56e13bcca50b1d7852ae484159a3909061951e8c59aaadbb4b071e02dd02d"}
Nov 03 00:09:37.413615 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:09:37.412231    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:09:40.227965 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:40.227858    5012 generic.go:356] "Generic (PLEG): container finished" podID="a276be2c-84f0-4457-91fa-d5dedacaa865" containerID="40b8eb6c59608761e95a97d3075e78e9effef1200703f2b3d477a4d55f679634" exitCode=0
Nov 03 00:09:40.228449 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:40.227986    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c" event={"ID":"a276be2c-84f0-4457-91fa-d5dedacaa865","Type":"ContainerDied","Data":"40b8eb6c59608761e95a97d3075e78e9effef1200703f2b3d477a4d55f679634"}
Nov 03 00:09:41.447977 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:41.447939    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c"
Nov 03 00:09:41.479898 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:41.478967    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a276be2c-84f0-4457-91fa-d5dedacaa865-kube-api-access-nnbhr\" (UniqueName: \"kubernetes.io/projected/a276be2c-84f0-4457-91fa-d5dedacaa865-kube-api-access-nnbhr\") pod \"a276be2c-84f0-4457-91fa-d5dedacaa865\" (UID: \"a276be2c-84f0-4457-91fa-d5dedacaa865\") "
Nov 03 00:09:41.479898 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:41.479029    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a276be2c-84f0-4457-91fa-d5dedacaa865-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a276be2c-84f0-4457-91fa-d5dedacaa865-bundle\") pod \"a276be2c-84f0-4457-91fa-d5dedacaa865\" (UID: \"a276be2c-84f0-4457-91fa-d5dedacaa865\") "
Nov 03 00:09:41.479898 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:41.479135    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a276be2c-84f0-4457-91fa-d5dedacaa865-util\" (UniqueName: \"kubernetes.io/empty-dir/a276be2c-84f0-4457-91fa-d5dedacaa865-util\") pod \"a276be2c-84f0-4457-91fa-d5dedacaa865\" (UID: \"a276be2c-84f0-4457-91fa-d5dedacaa865\") "
Nov 03 00:09:41.480879 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:41.480258    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a276be2c-84f0-4457-91fa-d5dedacaa865-bundle" pod "a276be2c-84f0-4457-91fa-d5dedacaa865" (UID: "a276be2c-84f0-4457-91fa-d5dedacaa865"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:09:41.486590 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:41.486552    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a276be2c-84f0-4457-91fa-d5dedacaa865-kube-api-access-nnbhr" pod "a276be2c-84f0-4457-91fa-d5dedacaa865" (UID: "a276be2c-84f0-4457-91fa-d5dedacaa865"). InnerVolumeSpecName "kube-api-access-nnbhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:09:41.489989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:41.489889    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a276be2c-84f0-4457-91fa-d5dedacaa865-util" pod "a276be2c-84f0-4457-91fa-d5dedacaa865" (UID: "a276be2c-84f0-4457-91fa-d5dedacaa865"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:09:41.584587 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:41.584557    5012 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a276be2c-84f0-4457-91fa-d5dedacaa865-util\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:41.584760 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:41.584590    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnbhr\" (UniqueName: \"kubernetes.io/projected/a276be2c-84f0-4457-91fa-d5dedacaa865-kube-api-access-nnbhr\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:41.584760 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:41.584605    5012 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a276be2c-84f0-4457-91fa-d5dedacaa865-bundle\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:42.246550 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:42.246470    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9" event={"ID":"15b7bc3d-86d7-4dc5-93d5-8f213d700093","Type":"ContainerStarted","Data":"dd11c945c6237f5383e14e36b7160fc932223ea8ef11637252cbafb15c49ab93"}
Nov 03 00:09:42.249823 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:42.249679    5012 generic.go:356] "Generic (PLEG): container finished" podID="49ec319e-e958-40d8-89de-c58335aeeea0" containerID="57a8a221f8bda1c881692703bd914da4b5308600ffdbe908c8edec2c3b143662" exitCode=0
Nov 03 00:09:42.249823 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:42.249766    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt" event={"ID":"49ec319e-e958-40d8-89de-c58335aeeea0","Type":"ContainerDied","Data":"57a8a221f8bda1c881692703bd914da4b5308600ffdbe908c8edec2c3b143662"}
Nov 03 00:09:42.252036 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:42.252010    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c" event={"ID":"a276be2c-84f0-4457-91fa-d5dedacaa865","Type":"ContainerDied","Data":"a64c63819a456fbb86b110d2fce1f49be15af26be9d32745a3d11febb4cd9e1a"}
Nov 03 00:09:42.252152 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:42.252053    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a64c63819a456fbb86b110d2fce1f49be15af26be9d32745a3d11febb4cd9e1a"
Nov 03 00:09:42.252152 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:42.252114    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptv7c"
Nov 03 00:09:42.262854 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:42.262248    5012 generic.go:356] "Generic (PLEG): container finished" podID="acb40932-dcac-4068-a88b-1479f7904f3c" containerID="8f103abb8ca77f9b7bd51c91a77aa71324c32bd593b05471df9bb075cde632e1" exitCode=0
Nov 03 00:09:42.262854 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:42.262622    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md" event={"ID":"acb40932-dcac-4068-a88b-1479f7904f3c","Type":"ContainerDied","Data":"8f103abb8ca77f9b7bd51c91a77aa71324c32bd593b05471df9bb075cde632e1"}
Nov 03 00:09:42.271663 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:42.271639    5012 generic.go:356] "Generic (PLEG): container finished" podID="47d9572f-cce3-40b5-a8e0-7a9fb348e127" containerID="679ba205e2b8c2d2acb5cef6e6b9c33567ea378ba0865f28ace8db152f679956" exitCode=0
Nov 03 00:09:42.271770 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:42.271751    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6" event={"ID":"47d9572f-cce3-40b5-a8e0-7a9fb348e127","Type":"ContainerDied","Data":"679ba205e2b8c2d2acb5cef6e6b9c33567ea378ba0865f28ace8db152f679956"}
Nov 03 00:09:42.275846 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:42.275821    5012 generic.go:356] "Generic (PLEG): container finished" podID="0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945" containerID="10e1878fdcae385b35a3f2f15d943fc8e286bc550e06a1404175af99433c67a6" exitCode=0
Nov 03 00:09:42.276053 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:42.276033    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j" event={"ID":"0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945","Type":"ContainerDied","Data":"10e1878fdcae385b35a3f2f15d943fc8e286bc550e06a1404175af99433c67a6"}
Nov 03 00:09:42.279482 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:42.279456    5012 generic.go:356] "Generic (PLEG): container finished" podID="96e7a1a7-b210-41d5-af65-4722b7b33e67" containerID="53dfd5340a04e8ea6f63e53975117f293da0026b041486aa7f6d0207059e3e8f" exitCode=0
Nov 03 00:09:42.279595 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:42.279547    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2" event={"ID":"96e7a1a7-b210-41d5-af65-4722b7b33e67","Type":"ContainerDied","Data":"53dfd5340a04e8ea6f63e53975117f293da0026b041486aa7f6d0207059e3e8f"}
Nov 03 00:09:42.283113 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:42.283087    5012 generic.go:356] "Generic (PLEG): container finished" podID="997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca" containerID="763cc45d2fe112046d9b2d08ef9c7e655f5d75278f8da4def11f2f65c14a7aea" exitCode=0
Nov 03 00:09:42.283207 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:42.283163    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv" event={"ID":"997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca","Type":"ContainerDied","Data":"763cc45d2fe112046d9b2d08ef9c7e655f5d75278f8da4def11f2f65c14a7aea"}
Nov 03 00:09:42.286307 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:42.286284    5012 generic.go:356] "Generic (PLEG): container finished" podID="251d8990-c450-4333-8dca-456dd9640e09" containerID="39f025504be4ed728d46a0221a284867f7b2f9064fde4e35b446eccba645e368" exitCode=0
Nov 03 00:09:42.286403 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:42.286351    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55" event={"ID":"251d8990-c450-4333-8dca-456dd9640e09","Type":"ContainerDied","Data":"39f025504be4ed728d46a0221a284867f7b2f9064fde4e35b446eccba645e368"}
Nov 03 00:09:42.289404 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:42.289327    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9" podStartSLOduration=2.766725815 podStartE2EDuration="51.289311626s" totalImagesPullingTime="48.522585811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:08:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:42.289079293 +0000 UTC m=+265.168663338" watchObservedRunningTime="2026-11-03 00:09:42.289311626 +0000 UTC m=+265.168895724"
Nov 03 00:09:43.296419 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.296336    5012 generic.go:356] "Generic (PLEG): container finished" podID="0158fe4a-e132-4d0e-92e9-8ce181c2c559" containerID="c256966dce85c95615b81962892fd01a59d1fb91b726dc19b06886c6b683c285" exitCode=0
Nov 03 00:09:43.296869 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.296509    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b" event={"ID":"0158fe4a-e132-4d0e-92e9-8ce181c2c559","Type":"ContainerDied","Data":"c256966dce85c95615b81962892fd01a59d1fb91b726dc19b06886c6b683c285"}
Nov 03 00:09:43.300468 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.299824    5012 generic.go:356] "Generic (PLEG): container finished" podID="15b7bc3d-86d7-4dc5-93d5-8f213d700093" containerID="dd11c945c6237f5383e14e36b7160fc932223ea8ef11637252cbafb15c49ab93" exitCode=0
Nov 03 00:09:43.300468 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.299964    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9" event={"ID":"15b7bc3d-86d7-4dc5-93d5-8f213d700093","Type":"ContainerDied","Data":"dd11c945c6237f5383e14e36b7160fc932223ea8ef11637252cbafb15c49ab93"}
Nov 03 00:09:43.510437 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.510218    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2"
Nov 03 00:09:43.576117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.576086    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv"
Nov 03 00:09:43.578788 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.578679    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j"
Nov 03 00:09:43.581732 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.581650    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6"
Nov 03 00:09:43.584270 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.584247    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt"
Nov 03 00:09:43.605185 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.605157    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca-kube-api-access-hrgd4\" (UniqueName: \"kubernetes.io/projected/997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca-kube-api-access-hrgd4\") pod \"997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca\" (UID: \"997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca\") "
Nov 03 00:09:43.605274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.605217    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d9572f-cce3-40b5-a8e0-7a9fb348e127-kube-api-access-6xkgw\" (UniqueName: \"kubernetes.io/projected/47d9572f-cce3-40b5-a8e0-7a9fb348e127-kube-api-access-6xkgw\") pod \"47d9572f-cce3-40b5-a8e0-7a9fb348e127\" (UID: \"47d9572f-cce3-40b5-a8e0-7a9fb348e127\") "
Nov 03 00:09:43.605320 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.605275    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945-bundle\") pod \"0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945\" (UID: \"0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945\") "
Nov 03 00:09:43.605320 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.605312    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945-util\" (UniqueName: \"kubernetes.io/empty-dir/0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945-util\") pod \"0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945\" (UID: \"0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945\") "
Nov 03 00:09:43.605385 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.605355    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d9572f-cce3-40b5-a8e0-7a9fb348e127-util\" (UniqueName: \"kubernetes.io/empty-dir/47d9572f-cce3-40b5-a8e0-7a9fb348e127-util\") pod \"47d9572f-cce3-40b5-a8e0-7a9fb348e127\" (UID: \"47d9572f-cce3-40b5-a8e0-7a9fb348e127\") "
Nov 03 00:09:43.605428 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.605414    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945-kube-api-access-6jpbw\" (UniqueName: \"kubernetes.io/projected/0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945-kube-api-access-6jpbw\") pod \"0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945\" (UID: \"0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945\") "
Nov 03 00:09:43.605482 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.605469    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d9572f-cce3-40b5-a8e0-7a9fb348e127-bundle\" (UniqueName: \"kubernetes.io/empty-dir/47d9572f-cce3-40b5-a8e0-7a9fb348e127-bundle\") pod \"47d9572f-cce3-40b5-a8e0-7a9fb348e127\" (UID: \"47d9572f-cce3-40b5-a8e0-7a9fb348e127\") "
Nov 03 00:09:43.605537 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.605503    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca-util\" (UniqueName: \"kubernetes.io/empty-dir/997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca-util\") pod \"997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca\" (UID: \"997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca\") "
Nov 03 00:09:43.605591 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.605534    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e7a1a7-b210-41d5-af65-4722b7b33e67-util\" (UniqueName: \"kubernetes.io/empty-dir/96e7a1a7-b210-41d5-af65-4722b7b33e67-util\") pod \"96e7a1a7-b210-41d5-af65-4722b7b33e67\" (UID: \"96e7a1a7-b210-41d5-af65-4722b7b33e67\") "
Nov 03 00:09:43.605591 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.605574    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e7a1a7-b210-41d5-af65-4722b7b33e67-bundle\" (UniqueName: \"kubernetes.io/empty-dir/96e7a1a7-b210-41d5-af65-4722b7b33e67-bundle\") pod \"96e7a1a7-b210-41d5-af65-4722b7b33e67\" (UID: \"96e7a1a7-b210-41d5-af65-4722b7b33e67\") "
Nov 03 00:09:43.605693 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.605605    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca-bundle\" (UniqueName: \"kubernetes.io/empty-dir/997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca-bundle\") pod \"997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca\" (UID: \"997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca\") "
Nov 03 00:09:43.605693 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.605640    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e7a1a7-b210-41d5-af65-4722b7b33e67-kube-api-access-kdbhl\" (UniqueName: \"kubernetes.io/projected/96e7a1a7-b210-41d5-af65-4722b7b33e67-kube-api-access-kdbhl\") pod \"96e7a1a7-b210-41d5-af65-4722b7b33e67\" (UID: \"96e7a1a7-b210-41d5-af65-4722b7b33e67\") "
Nov 03 00:09:43.606589 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.606556    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e7a1a7-b210-41d5-af65-4722b7b33e67-bundle" pod "96e7a1a7-b210-41d5-af65-4722b7b33e67" (UID: "96e7a1a7-b210-41d5-af65-4722b7b33e67"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:09:43.609483 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.609423    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca-bundle" pod "997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca" (UID: "997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:09:43.609782 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.609760    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d9572f-cce3-40b5-a8e0-7a9fb348e127-bundle" pod "47d9572f-cce3-40b5-a8e0-7a9fb348e127" (UID: "47d9572f-cce3-40b5-a8e0-7a9fb348e127"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:09:43.611690 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.611667    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945-util" pod "0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945" (UID: "0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:09:43.616229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.616196    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945-bundle" pod "0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945" (UID: "0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:09:43.618311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.618283    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d9572f-cce3-40b5-a8e0-7a9fb348e127-kube-api-access-6xkgw" pod "47d9572f-cce3-40b5-a8e0-7a9fb348e127" (UID: "47d9572f-cce3-40b5-a8e0-7a9fb348e127"). InnerVolumeSpecName "kube-api-access-6xkgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:09:43.618411 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.618329    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e7a1a7-b210-41d5-af65-4722b7b33e67-kube-api-access-kdbhl" pod "96e7a1a7-b210-41d5-af65-4722b7b33e67" (UID: "96e7a1a7-b210-41d5-af65-4722b7b33e67"). InnerVolumeSpecName "kube-api-access-kdbhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:09:43.618577 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.618552    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945-kube-api-access-6jpbw" pod "0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945" (UID: "0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945"). InnerVolumeSpecName "kube-api-access-6jpbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:09:43.618744 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.618720    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca-kube-api-access-hrgd4" pod "997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca" (UID: "997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca"). InnerVolumeSpecName "kube-api-access-hrgd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:09:43.623320 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.623288    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d9572f-cce3-40b5-a8e0-7a9fb348e127-util" pod "47d9572f-cce3-40b5-a8e0-7a9fb348e127" (UID: "47d9572f-cce3-40b5-a8e0-7a9fb348e127"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:09:43.625000 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.624975    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca-util" pod "997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca" (UID: "997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:09:43.625846 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.625824    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md"
Nov 03 00:09:43.625956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.625935    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e7a1a7-b210-41d5-af65-4722b7b33e67-util" pod "96e7a1a7-b210-41d5-af65-4722b7b33e67" (UID: "96e7a1a7-b210-41d5-af65-4722b7b33e67"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:09:43.633332 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.633310    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55"
Nov 03 00:09:43.706790 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.706756    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb40932-dcac-4068-a88b-1479f7904f3c-util\" (UniqueName: \"kubernetes.io/empty-dir/acb40932-dcac-4068-a88b-1479f7904f3c-util\") pod \"acb40932-dcac-4068-a88b-1479f7904f3c\" (UID: \"acb40932-dcac-4068-a88b-1479f7904f3c\") "
Nov 03 00:09:43.706992 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.706813    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49ec319e-e958-40d8-89de-c58335aeeea0-kube-api-access-rxrmq\" (UniqueName: \"kubernetes.io/projected/49ec319e-e958-40d8-89de-c58335aeeea0-kube-api-access-rxrmq\") pod \"49ec319e-e958-40d8-89de-c58335aeeea0\" (UID: \"49ec319e-e958-40d8-89de-c58335aeeea0\") "
Nov 03 00:09:43.706992 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.706871    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb40932-dcac-4068-a88b-1479f7904f3c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/acb40932-dcac-4068-a88b-1479f7904f3c-bundle\") pod \"acb40932-dcac-4068-a88b-1479f7904f3c\" (UID: \"acb40932-dcac-4068-a88b-1479f7904f3c\") "
Nov 03 00:09:43.706992 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.706928    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acb40932-dcac-4068-a88b-1479f7904f3c-kube-api-access-f5dkt\" (UniqueName: \"kubernetes.io/projected/acb40932-dcac-4068-a88b-1479f7904f3c-kube-api-access-f5dkt\") pod \"acb40932-dcac-4068-a88b-1479f7904f3c\" (UID: \"acb40932-dcac-4068-a88b-1479f7904f3c\") "
Nov 03 00:09:43.706992 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.706983    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251d8990-c450-4333-8dca-456dd9640e09-kube-api-access-q9tjx\" (UniqueName: \"kubernetes.io/projected/251d8990-c450-4333-8dca-456dd9640e09-kube-api-access-q9tjx\") pod \"251d8990-c450-4333-8dca-456dd9640e09\" (UID: \"251d8990-c450-4333-8dca-456dd9640e09\") "
Nov 03 00:09:43.707216 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.707029    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251d8990-c450-4333-8dca-456dd9640e09-util\" (UniqueName: \"kubernetes.io/empty-dir/251d8990-c450-4333-8dca-456dd9640e09-util\") pod \"251d8990-c450-4333-8dca-456dd9640e09\" (UID: \"251d8990-c450-4333-8dca-456dd9640e09\") "
Nov 03 00:09:43.707216 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.707068    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49ec319e-e958-40d8-89de-c58335aeeea0-util\" (UniqueName: \"kubernetes.io/empty-dir/49ec319e-e958-40d8-89de-c58335aeeea0-util\") pod \"49ec319e-e958-40d8-89de-c58335aeeea0\" (UID: \"49ec319e-e958-40d8-89de-c58335aeeea0\") "
Nov 03 00:09:43.707216 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.707098    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251d8990-c450-4333-8dca-456dd9640e09-bundle\" (UniqueName: \"kubernetes.io/empty-dir/251d8990-c450-4333-8dca-456dd9640e09-bundle\") pod \"251d8990-c450-4333-8dca-456dd9640e09\" (UID: \"251d8990-c450-4333-8dca-456dd9640e09\") "
Nov 03 00:09:43.707216 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.707140    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49ec319e-e958-40d8-89de-c58335aeeea0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/49ec319e-e958-40d8-89de-c58335aeeea0-bundle\") pod \"49ec319e-e958-40d8-89de-c58335aeeea0\" (UID: \"49ec319e-e958-40d8-89de-c58335aeeea0\") "
Nov 03 00:09:43.707409 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.707393    5012 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47d9572f-cce3-40b5-a8e0-7a9fb348e127-bundle\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:43.707465 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.707415    5012 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca-util\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:43.707465 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.707429    5012 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96e7a1a7-b210-41d5-af65-4722b7b33e67-util\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:43.707465 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.707444    5012 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96e7a1a7-b210-41d5-af65-4722b7b33e67-bundle\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:43.707465 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.707458    5012 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca-bundle\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:43.707650 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.707471    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdbhl\" (UniqueName: \"kubernetes.io/projected/96e7a1a7-b210-41d5-af65-4722b7b33e67-kube-api-access-kdbhl\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:43.707650 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.707489    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrgd4\" (UniqueName: \"kubernetes.io/projected/997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca-kube-api-access-hrgd4\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:43.707650 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.707503    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xkgw\" (UniqueName: \"kubernetes.io/projected/47d9572f-cce3-40b5-a8e0-7a9fb348e127-kube-api-access-6xkgw\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:43.707650 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.707518    5012 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945-bundle\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:43.707650 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.707531    5012 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945-util\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:43.707650 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.707545    5012 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47d9572f-cce3-40b5-a8e0-7a9fb348e127-util\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:43.707650 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.707558    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jpbw\" (UniqueName: \"kubernetes.io/projected/0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945-kube-api-access-6jpbw\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:43.708973 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.708700    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb40932-dcac-4068-a88b-1479f7904f3c-bundle" pod "acb40932-dcac-4068-a88b-1479f7904f3c" (UID: "acb40932-dcac-4068-a88b-1479f7904f3c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:09:43.709969 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.709940    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251d8990-c450-4333-8dca-456dd9640e09-bundle" pod "251d8990-c450-4333-8dca-456dd9640e09" (UID: "251d8990-c450-4333-8dca-456dd9640e09"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:09:43.710496 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.710467    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ec319e-e958-40d8-89de-c58335aeeea0-bundle" pod "49ec319e-e958-40d8-89de-c58335aeeea0" (UID: "49ec319e-e958-40d8-89de-c58335aeeea0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:09:43.711568 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.711540    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251d8990-c450-4333-8dca-456dd9640e09-kube-api-access-q9tjx" pod "251d8990-c450-4333-8dca-456dd9640e09" (UID: "251d8990-c450-4333-8dca-456dd9640e09"). InnerVolumeSpecName "kube-api-access-q9tjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:09:43.712828 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.712784    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb40932-dcac-4068-a88b-1479f7904f3c-kube-api-access-f5dkt" pod "acb40932-dcac-4068-a88b-1479f7904f3c" (UID: "acb40932-dcac-4068-a88b-1479f7904f3c"). InnerVolumeSpecName "kube-api-access-f5dkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:09:43.714211 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.714182    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ec319e-e958-40d8-89de-c58335aeeea0-kube-api-access-rxrmq" pod "49ec319e-e958-40d8-89de-c58335aeeea0" (UID: "49ec319e-e958-40d8-89de-c58335aeeea0"). InnerVolumeSpecName "kube-api-access-rxrmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:09:43.715484 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.715395    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ec319e-e958-40d8-89de-c58335aeeea0-util" pod "49ec319e-e958-40d8-89de-c58335aeeea0" (UID: "49ec319e-e958-40d8-89de-c58335aeeea0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:09:43.717273 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.717250    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251d8990-c450-4333-8dca-456dd9640e09-util" pod "251d8990-c450-4333-8dca-456dd9640e09" (UID: "251d8990-c450-4333-8dca-456dd9640e09"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:09:43.717760 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.717720    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb40932-dcac-4068-a88b-1479f7904f3c-util" pod "acb40932-dcac-4068-a88b-1479f7904f3c" (UID: "acb40932-dcac-4068-a88b-1479f7904f3c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:09:43.808578 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.808503    5012 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49ec319e-e958-40d8-89de-c58335aeeea0-util\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:43.808578 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.808531    5012 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/251d8990-c450-4333-8dca-456dd9640e09-bundle\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:43.808578 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.808540    5012 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49ec319e-e958-40d8-89de-c58335aeeea0-bundle\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:43.808578 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.808550    5012 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acb40932-dcac-4068-a88b-1479f7904f3c-util\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:43.808578 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.808561    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxrmq\" (UniqueName: \"kubernetes.io/projected/49ec319e-e958-40d8-89de-c58335aeeea0-kube-api-access-rxrmq\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:43.808578 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.808574    5012 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acb40932-dcac-4068-a88b-1479f7904f3c-bundle\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:43.808578 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.808583    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5dkt\" (UniqueName: \"kubernetes.io/projected/acb40932-dcac-4068-a88b-1479f7904f3c-kube-api-access-f5dkt\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:43.809034 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.808592    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9tjx\" (UniqueName: \"kubernetes.io/projected/251d8990-c450-4333-8dca-456dd9640e09-kube-api-access-q9tjx\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:43.809034 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:43.808600    5012 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/251d8990-c450-4333-8dca-456dd9640e09-util\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:44.306465 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.306435    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6"
Nov 03 00:09:44.306465 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.306450    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rrqp6" event={"ID":"47d9572f-cce3-40b5-a8e0-7a9fb348e127","Type":"ContainerDied","Data":"3164f1e22a2c6b44902b53d252c9fc86ca07e2a6f5c2bb163494f75320dedd66"}
Nov 03 00:09:44.306881 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.306500    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3164f1e22a2c6b44902b53d252c9fc86ca07e2a6f5c2bb163494f75320dedd66"
Nov 03 00:09:44.308551 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.308526    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j" event={"ID":"0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945","Type":"ContainerDied","Data":"8399f7509e9def53946c3f40d9139259518f01e583ccc797dccfed5b132bb918"}
Nov 03 00:09:44.308681 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.308571    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8399f7509e9def53946c3f40d9139259518f01e583ccc797dccfed5b132bb918"
Nov 03 00:09:44.308681 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.308536    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jhn4j"
Nov 03 00:09:44.310865 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.310840    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2"
Nov 03 00:09:44.310865 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.310847    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68g4r2" event={"ID":"96e7a1a7-b210-41d5-af65-4722b7b33e67","Type":"ContainerDied","Data":"df7f195f0218436598ca29caf65711f6ce78dd610141b9be92e73dcd61e9bfdf"}
Nov 03 00:09:44.311113 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.310893    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df7f195f0218436598ca29caf65711f6ce78dd610141b9be92e73dcd61e9bfdf"
Nov 03 00:09:44.317703 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.317683    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg" event={"ID":"07a769f1-c30f-4832-8159-fd6108718ffd","Type":"ContainerStarted","Data":"4ca0b435f9052b977093ce10f5d730d1dcab14fb50528c3963622833357b230c"}
Nov 03 00:09:44.319730 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.319708    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv"
Nov 03 00:09:44.319834 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.319709    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a75bsv" event={"ID":"997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca","Type":"ContainerDied","Data":"c37026178c095545b0ad4e707990e2f335cce81b3cfe69e93dfcf2d5e86ffffe"}
Nov 03 00:09:44.319885 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.319850    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c37026178c095545b0ad4e707990e2f335cce81b3cfe69e93dfcf2d5e86ffffe"
Nov 03 00:09:44.321970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.321948    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55" event={"ID":"251d8990-c450-4333-8dca-456dd9640e09","Type":"ContainerDied","Data":"00934c965ece9bf846ca745096e799e8cc4c68bfa030f83cd96b5d4b099c0795"}
Nov 03 00:09:44.322071 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.321981    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00934c965ece9bf846ca745096e799e8cc4c68bfa030f83cd96b5d4b099c0795"
Nov 03 00:09:44.322071 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.321955    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c28h55"
Nov 03 00:09:44.331452 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.325390    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms" event={"ID":"9c79292b-876b-4574-9238-2f29d04926fb","Type":"ContainerStarted","Data":"19bfacb52b52e8aaa7ec2de55efe948750200a928401aa78816477eaa113b91f"}
Nov 03 00:09:44.331452 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.327861    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt" event={"ID":"49ec319e-e958-40d8-89de-c58335aeeea0","Type":"ContainerDied","Data":"ed178dfd57575d89b0fab846152d5a7a215da75e652ccce97138022c423c4386"}
Nov 03 00:09:44.331452 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.327903    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed178dfd57575d89b0fab846152d5a7a215da75e652ccce97138022c423c4386"
Nov 03 00:09:44.331452 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.328049    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bv2kt"
Nov 03 00:09:44.331452 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.330444    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md" event={"ID":"acb40932-dcac-4068-a88b-1479f7904f3c","Type":"ContainerDied","Data":"f9dae51b9f74192925ac514132dcb4dcf777ee74af929013ca57c2707fcb1b7d"}
Nov 03 00:09:44.331452 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.330484    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9dae51b9f74192925ac514132dcb4dcf777ee74af929013ca57c2707fcb1b7d"
Nov 03 00:09:44.331452 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.330728    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71l75md"
Nov 03 00:09:44.363459 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.362749    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms" podStartSLOduration=2.800015598 podStartE2EDuration="49.36273674s" totalImagesPullingTime="46.562721142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:08:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:44.36265145 +0000 UTC m=+267.242235495" watchObservedRunningTime="2026-11-03 00:09:44.36273674 +0000 UTC m=+267.242320783"
Nov 03 00:09:44.408827 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.408780    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg" podStartSLOduration=3.078188308 podStartE2EDuration="52.408764798s" totalImagesPullingTime="49.33057649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:08:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:44.407933815 +0000 UTC m=+267.287517852" watchObservedRunningTime="2026-11-03 00:09:44.408764798 +0000 UTC m=+267.288348839"
Nov 03 00:09:44.514113 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.514088    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b"
Nov 03 00:09:44.516685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.516669    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9"
Nov 03 00:09:44.614323 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.614302    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0158fe4a-e132-4d0e-92e9-8ce181c2c559-kube-api-access-bmfv6\" (UniqueName: \"kubernetes.io/projected/0158fe4a-e132-4d0e-92e9-8ce181c2c559-kube-api-access-bmfv6\") pod \"0158fe4a-e132-4d0e-92e9-8ce181c2c559\" (UID: \"0158fe4a-e132-4d0e-92e9-8ce181c2c559\") "
Nov 03 00:09:44.614449 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.614337    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b7bc3d-86d7-4dc5-93d5-8f213d700093-bundle\" (UniqueName: \"kubernetes.io/empty-dir/15b7bc3d-86d7-4dc5-93d5-8f213d700093-bundle\") pod \"15b7bc3d-86d7-4dc5-93d5-8f213d700093\" (UID: \"15b7bc3d-86d7-4dc5-93d5-8f213d700093\") "
Nov 03 00:09:44.614449 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.614382    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b7bc3d-86d7-4dc5-93d5-8f213d700093-util\" (UniqueName: \"kubernetes.io/empty-dir/15b7bc3d-86d7-4dc5-93d5-8f213d700093-util\") pod \"15b7bc3d-86d7-4dc5-93d5-8f213d700093\" (UID: \"15b7bc3d-86d7-4dc5-93d5-8f213d700093\") "
Nov 03 00:09:44.614449 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.614423    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0158fe4a-e132-4d0e-92e9-8ce181c2c559-util\" (UniqueName: \"kubernetes.io/empty-dir/0158fe4a-e132-4d0e-92e9-8ce181c2c559-util\") pod \"0158fe4a-e132-4d0e-92e9-8ce181c2c559\" (UID: \"0158fe4a-e132-4d0e-92e9-8ce181c2c559\") "
Nov 03 00:09:44.614612 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.614464    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15b7bc3d-86d7-4dc5-93d5-8f213d700093-kube-api-access-bxfgp\" (UniqueName: \"kubernetes.io/projected/15b7bc3d-86d7-4dc5-93d5-8f213d700093-kube-api-access-bxfgp\") pod \"15b7bc3d-86d7-4dc5-93d5-8f213d700093\" (UID: \"15b7bc3d-86d7-4dc5-93d5-8f213d700093\") "
Nov 03 00:09:44.614612 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.614507    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0158fe4a-e132-4d0e-92e9-8ce181c2c559-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0158fe4a-e132-4d0e-92e9-8ce181c2c559-bundle\") pod \"0158fe4a-e132-4d0e-92e9-8ce181c2c559\" (UID: \"0158fe4a-e132-4d0e-92e9-8ce181c2c559\") "
Nov 03 00:09:44.614990 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.614964    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b7bc3d-86d7-4dc5-93d5-8f213d700093-bundle" pod "15b7bc3d-86d7-4dc5-93d5-8f213d700093" (UID: "15b7bc3d-86d7-4dc5-93d5-8f213d700093"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:09:44.615396 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.615368    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0158fe4a-e132-4d0e-92e9-8ce181c2c559-bundle" pod "0158fe4a-e132-4d0e-92e9-8ce181c2c559" (UID: "0158fe4a-e132-4d0e-92e9-8ce181c2c559"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:09:44.616440 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.616413    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0158fe4a-e132-4d0e-92e9-8ce181c2c559-kube-api-access-bmfv6" pod "0158fe4a-e132-4d0e-92e9-8ce181c2c559" (UID: "0158fe4a-e132-4d0e-92e9-8ce181c2c559"). InnerVolumeSpecName "kube-api-access-bmfv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:09:44.616838 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.616820    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b7bc3d-86d7-4dc5-93d5-8f213d700093-kube-api-access-bxfgp" pod "15b7bc3d-86d7-4dc5-93d5-8f213d700093" (UID: "15b7bc3d-86d7-4dc5-93d5-8f213d700093"). InnerVolumeSpecName "kube-api-access-bxfgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:09:44.619741 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.619719    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0158fe4a-e132-4d0e-92e9-8ce181c2c559-util" pod "0158fe4a-e132-4d0e-92e9-8ce181c2c559" (UID: "0158fe4a-e132-4d0e-92e9-8ce181c2c559"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:09:44.620313 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.620292    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b7bc3d-86d7-4dc5-93d5-8f213d700093-util" pod "15b7bc3d-86d7-4dc5-93d5-8f213d700093" (UID: "15b7bc3d-86d7-4dc5-93d5-8f213d700093"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:09:44.715091 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.715069    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmfv6\" (UniqueName: \"kubernetes.io/projected/0158fe4a-e132-4d0e-92e9-8ce181c2c559-kube-api-access-bmfv6\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:44.715091 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.715090    5012 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15b7bc3d-86d7-4dc5-93d5-8f213d700093-bundle\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:44.715226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.715099    5012 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15b7bc3d-86d7-4dc5-93d5-8f213d700093-util\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:44.715226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.715107    5012 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0158fe4a-e132-4d0e-92e9-8ce181c2c559-util\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:44.715226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.715115    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxfgp\" (UniqueName: \"kubernetes.io/projected/15b7bc3d-86d7-4dc5-93d5-8f213d700093-kube-api-access-bxfgp\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:44.715226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:44.715123    5012 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0158fe4a-e132-4d0e-92e9-8ce181c2c559-bundle\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:45.336566 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:45.336541    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9"
Nov 03 00:09:45.336944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:45.336540    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpdw9" event={"ID":"15b7bc3d-86d7-4dc5-93d5-8f213d700093","Type":"ContainerDied","Data":"4b6383d259e1b59103bdb14c0dbaaa0d3e1a00f0ea9aad2bc91ac98abcd4b6bb"}
Nov 03 00:09:45.336944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:45.336697    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b6383d259e1b59103bdb14c0dbaaa0d3e1a00f0ea9aad2bc91ac98abcd4b6bb"
Nov 03 00:09:45.339014 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:45.338989    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b" event={"ID":"0158fe4a-e132-4d0e-92e9-8ce181c2c559","Type":"ContainerDied","Data":"4db1f60539ef1edffced89272827b24367ce3b9c4617a5d1aea205b0b9850bc7"}
Nov 03 00:09:45.339014 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:45.339006    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvjd7b"
Nov 03 00:09:45.339199 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:45.339031    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4db1f60539ef1edffced89272827b24367ce3b9c4617a5d1aea205b0b9850bc7"
Nov 03 00:09:45.341380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:45.341358    5012 generic.go:356] "Generic (PLEG): container finished" podID="07a769f1-c30f-4832-8159-fd6108718ffd" containerID="4ca0b435f9052b977093ce10f5d730d1dcab14fb50528c3963622833357b230c" exitCode=0
Nov 03 00:09:45.341466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:45.341440    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg" event={"ID":"07a769f1-c30f-4832-8159-fd6108718ffd","Type":"ContainerDied","Data":"4ca0b435f9052b977093ce10f5d730d1dcab14fb50528c3963622833357b230c"}
Nov 03 00:09:45.343501 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:45.343481    5012 generic.go:356] "Generic (PLEG): container finished" podID="9c79292b-876b-4574-9238-2f29d04926fb" containerID="19bfacb52b52e8aaa7ec2de55efe948750200a928401aa78816477eaa113b91f" exitCode=0
Nov 03 00:09:45.343579 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:45.343512    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms" event={"ID":"9c79292b-876b-4574-9238-2f29d04926fb","Type":"ContainerDied","Data":"19bfacb52b52e8aaa7ec2de55efe948750200a928401aa78816477eaa113b91f"}
Nov 03 00:09:46.473957 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:46.473935    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg"
Nov 03 00:09:46.501584 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:46.501561    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms"
Nov 03 00:09:46.527902 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:46.527875    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c79292b-876b-4574-9238-2f29d04926fb-kube-api-access-r922g\" (UniqueName: \"kubernetes.io/projected/9c79292b-876b-4574-9238-2f29d04926fb-kube-api-access-r922g\") pod \"9c79292b-876b-4574-9238-2f29d04926fb\" (UID: \"9c79292b-876b-4574-9238-2f29d04926fb\") "
Nov 03 00:09:46.528023 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:46.527953    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a769f1-c30f-4832-8159-fd6108718ffd-util\" (UniqueName: \"kubernetes.io/empty-dir/07a769f1-c30f-4832-8159-fd6108718ffd-util\") pod \"07a769f1-c30f-4832-8159-fd6108718ffd\" (UID: \"07a769f1-c30f-4832-8159-fd6108718ffd\") "
Nov 03 00:09:46.528079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:46.528040    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a769f1-c30f-4832-8159-fd6108718ffd-kube-api-access-5dvdm\" (UniqueName: \"kubernetes.io/projected/07a769f1-c30f-4832-8159-fd6108718ffd-kube-api-access-5dvdm\") pod \"07a769f1-c30f-4832-8159-fd6108718ffd\" (UID: \"07a769f1-c30f-4832-8159-fd6108718ffd\") "
Nov 03 00:09:46.528115 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:46.528086    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a769f1-c30f-4832-8159-fd6108718ffd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/07a769f1-c30f-4832-8159-fd6108718ffd-bundle\") pod \"07a769f1-c30f-4832-8159-fd6108718ffd\" (UID: \"07a769f1-c30f-4832-8159-fd6108718ffd\") "
Nov 03 00:09:46.528154 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:46.528142    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c79292b-876b-4574-9238-2f29d04926fb-util\" (UniqueName: \"kubernetes.io/empty-dir/9c79292b-876b-4574-9238-2f29d04926fb-util\") pod \"9c79292b-876b-4574-9238-2f29d04926fb\" (UID: \"9c79292b-876b-4574-9238-2f29d04926fb\") "
Nov 03 00:09:46.528188 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:46.528171    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c79292b-876b-4574-9238-2f29d04926fb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c79292b-876b-4574-9238-2f29d04926fb-bundle\") pod \"9c79292b-876b-4574-9238-2f29d04926fb\" (UID: \"9c79292b-876b-4574-9238-2f29d04926fb\") "
Nov 03 00:09:46.528557 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:46.528527    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a769f1-c30f-4832-8159-fd6108718ffd-bundle" pod "07a769f1-c30f-4832-8159-fd6108718ffd" (UID: "07a769f1-c30f-4832-8159-fd6108718ffd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:09:46.528837 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:46.528816    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c79292b-876b-4574-9238-2f29d04926fb-bundle" pod "9c79292b-876b-4574-9238-2f29d04926fb" (UID: "9c79292b-876b-4574-9238-2f29d04926fb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:09:46.530412 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:46.530387    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c79292b-876b-4574-9238-2f29d04926fb-kube-api-access-r922g" pod "9c79292b-876b-4574-9238-2f29d04926fb" (UID: "9c79292b-876b-4574-9238-2f29d04926fb"). InnerVolumeSpecName "kube-api-access-r922g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:09:46.530524 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:46.530492    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a769f1-c30f-4832-8159-fd6108718ffd-kube-api-access-5dvdm" pod "07a769f1-c30f-4832-8159-fd6108718ffd" (UID: "07a769f1-c30f-4832-8159-fd6108718ffd"). InnerVolumeSpecName "kube-api-access-5dvdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:09:46.534995 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:46.534967    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c79292b-876b-4574-9238-2f29d04926fb-util" pod "9c79292b-876b-4574-9238-2f29d04926fb" (UID: "9c79292b-876b-4574-9238-2f29d04926fb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:09:46.536887 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:46.536864    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a769f1-c30f-4832-8159-fd6108718ffd-util" pod "07a769f1-c30f-4832-8159-fd6108718ffd" (UID: "07a769f1-c30f-4832-8159-fd6108718ffd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:09:46.629385 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:46.629324    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dvdm\" (UniqueName: \"kubernetes.io/projected/07a769f1-c30f-4832-8159-fd6108718ffd-kube-api-access-5dvdm\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:46.629385 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:46.629348    5012 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07a769f1-c30f-4832-8159-fd6108718ffd-bundle\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:46.629385 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:46.629359    5012 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c79292b-876b-4574-9238-2f29d04926fb-util\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:46.629385 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:46.629366    5012 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c79292b-876b-4574-9238-2f29d04926fb-bundle\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:46.629385 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:46.629375    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r922g\" (UniqueName: \"kubernetes.io/projected/9c79292b-876b-4574-9238-2f29d04926fb-kube-api-access-r922g\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:46.629385 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:46.629383    5012 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07a769f1-c30f-4832-8159-fd6108718ffd-util\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:09:47.354395 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:47.354367    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms"
Nov 03 00:09:47.354535 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:47.354392    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs4qms" event={"ID":"9c79292b-876b-4574-9238-2f29d04926fb","Type":"ContainerDied","Data":"0439b15a3f7855638b0842367e2775436a674718193378724e08a3d9290a160a"}
Nov 03 00:09:47.354535 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:47.354468    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0439b15a3f7855638b0842367e2775436a674718193378724e08a3d9290a160a"
Nov 03 00:09:47.356594 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:47.356547    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg" event={"ID":"07a769f1-c30f-4832-8159-fd6108718ffd","Type":"ContainerDied","Data":"94e95eb61bdf261c3fc9728e9f9803fd83fe36cf219eff1de15ecee7c4335bc5"}
Nov 03 00:09:47.356594 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:47.356590    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94e95eb61bdf261c3fc9728e9f9803fd83fe36cf219eff1de15ecee7c4335bc5"
Nov 03 00:09:47.356768 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:47.356600    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445qfmg"
Nov 03 00:09:52.096196 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096162    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-6mmrg"]
Nov 03 00:09:52.096627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096490    5012 cpu_manager.go:414] "containerMap: removing container" podUID="07a769f1-c30f-4832-8159-fd6108718ffd" containerName="util"
Nov 03 00:09:52.096627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096502    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a769f1-c30f-4832-8159-fd6108718ffd" containerName="util"
Nov 03 00:09:52.096627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096513    5012 cpu_manager.go:414] "containerMap: removing container" podUID="0158fe4a-e132-4d0e-92e9-8ce181c2c559" containerName="extract"
Nov 03 00:09:52.096627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096518    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0158fe4a-e132-4d0e-92e9-8ce181c2c559" containerName="extract"
Nov 03 00:09:52.096627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096526    5012 cpu_manager.go:414] "containerMap: removing container" podUID="96e7a1a7-b210-41d5-af65-4722b7b33e67" containerName="pull"
Nov 03 00:09:52.096627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096532    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e7a1a7-b210-41d5-af65-4722b7b33e67" containerName="pull"
Nov 03 00:09:52.096627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096542    5012 cpu_manager.go:414] "containerMap: removing container" podUID="07a769f1-c30f-4832-8159-fd6108718ffd" containerName="extract"
Nov 03 00:09:52.096627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096551    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a769f1-c30f-4832-8159-fd6108718ffd" containerName="extract"
Nov 03 00:09:52.096627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096559    5012 cpu_manager.go:414] "containerMap: removing container" podUID="0158fe4a-e132-4d0e-92e9-8ce181c2c559" containerName="util"
Nov 03 00:09:52.096627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096563    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0158fe4a-e132-4d0e-92e9-8ce181c2c559" containerName="util"
Nov 03 00:09:52.096627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096569    5012 cpu_manager.go:414] "containerMap: removing container" podUID="a276be2c-84f0-4457-91fa-d5dedacaa865" containerName="util"
Nov 03 00:09:52.096627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096574    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a276be2c-84f0-4457-91fa-d5dedacaa865" containerName="util"
Nov 03 00:09:52.096627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096580    5012 cpu_manager.go:414] "containerMap: removing container" podUID="49ec319e-e958-40d8-89de-c58335aeeea0" containerName="pull"
Nov 03 00:09:52.096627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096585    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ec319e-e958-40d8-89de-c58335aeeea0" containerName="pull"
Nov 03 00:09:52.096627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096591    5012 cpu_manager.go:414] "containerMap: removing container" podUID="47d9572f-cce3-40b5-a8e0-7a9fb348e127" containerName="extract"
Nov 03 00:09:52.096627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096596    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47d9572f-cce3-40b5-a8e0-7a9fb348e127" containerName="extract"
Nov 03 00:09:52.096627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096602    5012 cpu_manager.go:414] "containerMap: removing container" podUID="0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945" containerName="util"
Nov 03 00:09:52.096627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096606    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945" containerName="util"
Nov 03 00:09:52.096627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096613    5012 cpu_manager.go:414] "containerMap: removing container" podUID="251d8990-c450-4333-8dca-456dd9640e09" containerName="pull"
Nov 03 00:09:52.096627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096619    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251d8990-c450-4333-8dca-456dd9640e09" containerName="pull"
Nov 03 00:09:52.096627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096629    5012 cpu_manager.go:414] "containerMap: removing container" podUID="251d8990-c450-4333-8dca-456dd9640e09" containerName="extract"
Nov 03 00:09:52.096627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096634    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251d8990-c450-4333-8dca-456dd9640e09" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096640    5012 cpu_manager.go:414] "containerMap: removing container" podUID="acb40932-dcac-4068-a88b-1479f7904f3c" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096644    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acb40932-dcac-4068-a88b-1479f7904f3c" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096651    5012 cpu_manager.go:414] "containerMap: removing container" podUID="47d9572f-cce3-40b5-a8e0-7a9fb348e127" containerName="pull"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096655    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47d9572f-cce3-40b5-a8e0-7a9fb348e127" containerName="pull"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096661    5012 cpu_manager.go:414] "containerMap: removing container" podUID="acb40932-dcac-4068-a88b-1479f7904f3c" containerName="pull"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096665    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acb40932-dcac-4068-a88b-1479f7904f3c" containerName="pull"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096671    5012 cpu_manager.go:414] "containerMap: removing container" podUID="0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945" containerName="pull"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096675    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945" containerName="pull"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096680    5012 cpu_manager.go:414] "containerMap: removing container" podUID="15b7bc3d-86d7-4dc5-93d5-8f213d700093" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096684    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b7bc3d-86d7-4dc5-93d5-8f213d700093" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096693    5012 cpu_manager.go:414] "containerMap: removing container" podUID="9c79292b-876b-4574-9238-2f29d04926fb" containerName="util"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096700    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c79292b-876b-4574-9238-2f29d04926fb" containerName="util"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096711    5012 cpu_manager.go:414] "containerMap: removing container" podUID="997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca" containerName="pull"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096715    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca" containerName="pull"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096724    5012 cpu_manager.go:414] "containerMap: removing container" podUID="251d8990-c450-4333-8dca-456dd9640e09" containerName="util"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096728    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251d8990-c450-4333-8dca-456dd9640e09" containerName="util"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096734    5012 cpu_manager.go:414] "containerMap: removing container" podUID="997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096738    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096745    5012 cpu_manager.go:414] "containerMap: removing container" podUID="96e7a1a7-b210-41d5-af65-4722b7b33e67" containerName="util"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096749    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e7a1a7-b210-41d5-af65-4722b7b33e67" containerName="util"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096755    5012 cpu_manager.go:414] "containerMap: removing container" podUID="9c79292b-876b-4574-9238-2f29d04926fb" containerName="pull"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096759    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c79292b-876b-4574-9238-2f29d04926fb" containerName="pull"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096767    5012 cpu_manager.go:414] "containerMap: removing container" podUID="96e7a1a7-b210-41d5-af65-4722b7b33e67" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096773    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e7a1a7-b210-41d5-af65-4722b7b33e67" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096784    5012 cpu_manager.go:414] "containerMap: removing container" podUID="47d9572f-cce3-40b5-a8e0-7a9fb348e127" containerName="util"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096789    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47d9572f-cce3-40b5-a8e0-7a9fb348e127" containerName="util"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096795    5012 cpu_manager.go:414] "containerMap: removing container" podUID="acb40932-dcac-4068-a88b-1479f7904f3c" containerName="util"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096800    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acb40932-dcac-4068-a88b-1479f7904f3c" containerName="util"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096806    5012 cpu_manager.go:414] "containerMap: removing container" podUID="15b7bc3d-86d7-4dc5-93d5-8f213d700093" containerName="pull"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096810    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b7bc3d-86d7-4dc5-93d5-8f213d700093" containerName="pull"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096818    5012 cpu_manager.go:414] "containerMap: removing container" podUID="997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca" containerName="util"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096822    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca" containerName="util"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096829    5012 cpu_manager.go:414] "containerMap: removing container" podUID="15b7bc3d-86d7-4dc5-93d5-8f213d700093" containerName="util"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096833    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b7bc3d-86d7-4dc5-93d5-8f213d700093" containerName="util"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096839    5012 cpu_manager.go:414] "containerMap: removing container" podUID="a276be2c-84f0-4457-91fa-d5dedacaa865" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096844    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a276be2c-84f0-4457-91fa-d5dedacaa865" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096853    5012 cpu_manager.go:414] "containerMap: removing container" podUID="49ec319e-e958-40d8-89de-c58335aeeea0" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096859    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ec319e-e958-40d8-89de-c58335aeeea0" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096865    5012 cpu_manager.go:414] "containerMap: removing container" podUID="49ec319e-e958-40d8-89de-c58335aeeea0" containerName="util"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096869    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ec319e-e958-40d8-89de-c58335aeeea0" containerName="util"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096875    5012 cpu_manager.go:414] "containerMap: removing container" podUID="9c79292b-876b-4574-9238-2f29d04926fb" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096879    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c79292b-876b-4574-9238-2f29d04926fb" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096885    5012 cpu_manager.go:414] "containerMap: removing container" podUID="a276be2c-84f0-4457-91fa-d5dedacaa865" containerName="pull"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096889    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a276be2c-84f0-4457-91fa-d5dedacaa865" containerName="pull"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096895    5012 cpu_manager.go:414] "containerMap: removing container" podUID="07a769f1-c30f-4832-8159-fd6108718ffd" containerName="pull"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096899    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a769f1-c30f-4832-8159-fd6108718ffd" containerName="pull"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096921    5012 cpu_manager.go:414] "containerMap: removing container" podUID="0158fe4a-e132-4d0e-92e9-8ce181c2c559" containerName="pull"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096928    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0158fe4a-e132-4d0e-92e9-8ce181c2c559" containerName="pull"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096934    5012 cpu_manager.go:414] "containerMap: removing container" podUID="0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.096938    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.097012    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="47d9572f-cce3-40b5-a8e0-7a9fb348e127" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.097021    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="251d8990-c450-4333-8dca-456dd9640e09" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.097027    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="acb40932-dcac-4068-a88b-1479f7904f3c" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.097033    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b7bc3d-86d7-4dc5-93d5-8f213d700093" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.097038    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e7a1a7-b210-41d5-af65-4722b7b33e67" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.097045    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="997c7e6a-b7f7-4dcb-8e72-a0e6c90c34ca" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.097051    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="a276be2c-84f0-4457-91fa-d5dedacaa865" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.097056    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="49ec319e-e958-40d8-89de-c58335aeeea0" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.097062    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c79292b-876b-4574-9238-2f29d04926fb" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.097070    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fc4c17e-cf25-44d6-b8e4-9d6ebcd5e945" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.097080    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="0158fe4a-e132-4d0e-92e9-8ce181c2c559" containerName="extract"
Nov 03 00:09:52.097264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.097087    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a769f1-c30f-4832-8159-fd6108718ffd" containerName="extract"
Nov 03 00:09:52.101038 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.101022    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6mmrg"
Nov 03 00:09:52.127079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.127054    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 00:09:52.127209 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.127087    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-2w8vf\""
Nov 03 00:09:52.127209 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.127095    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 00:09:52.129608 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.129586    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-6mmrg"]
Nov 03 00:09:52.169284 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.169260    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgxrx\" (UniqueName: \"kubernetes.io/projected/56d60aa9-02ef-45c9-be0a-37f090753315-kube-api-access-jgxrx\") pod \"limitador-operator-controller-manager-5778bb4449-6mmrg\" (UID: \"56d60aa9-02ef-45c9-be0a-37f090753315\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6mmrg"
Nov 03 00:09:52.270208 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.270181    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgxrx\" (UniqueName: \"kubernetes.io/projected/56d60aa9-02ef-45c9-be0a-37f090753315-kube-api-access-jgxrx\") pod \"limitador-operator-controller-manager-5778bb4449-6mmrg\" (UID: \"56d60aa9-02ef-45c9-be0a-37f090753315\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6mmrg"
Nov 03 00:09:52.288254 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.288233    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgxrx\" (UniqueName: \"kubernetes.io/projected/56d60aa9-02ef-45c9-be0a-37f090753315-kube-api-access-jgxrx\") pod \"limitador-operator-controller-manager-5778bb4449-6mmrg\" (UID: \"56d60aa9-02ef-45c9-be0a-37f090753315\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6mmrg"
Nov 03 00:09:52.413289 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.413265    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6mmrg"
Nov 03 00:09:52.712870 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:52.712843    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-6mmrg"]
Nov 03 00:09:52.713772 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:09:52.713748    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d60aa9_02ef_45c9_be0a_37f090753315.slice/crio-fb69152d25751c07b5de0a41b43dff8e3b3bf58610c84abd16744aa2a1e365e5 WatchSource:0}: Error finding container fb69152d25751c07b5de0a41b43dff8e3b3bf58610c84abd16744aa2a1e365e5: Status 404 returned error can't find the container with id fb69152d25751c07b5de0a41b43dff8e3b3bf58610c84abd16744aa2a1e365e5
Nov 03 00:09:53.386585 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:53.386548    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6mmrg" event={"ID":"56d60aa9-02ef-45c9-be0a-37f090753315","Type":"ContainerStarted","Data":"fb69152d25751c07b5de0a41b43dff8e3b3bf58610c84abd16744aa2a1e365e5"}
Nov 03 00:09:56.214781 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.214743    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898"]
Nov 03 00:09:56.218209 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.218192    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898"
Nov 03 00:09:56.231512 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.231492    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 00:09:56.241172 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.241147    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 00:09:56.241172 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.241162    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 00:09:56.241308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.241175    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 00:09:56.241308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.241177    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 00:09:56.241308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.241190    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-kvgpd\""
Nov 03 00:09:56.257802 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.257779    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898"]
Nov 03 00:09:56.306007 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.305978    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6599f4db-ad1e-4234-bf93-dcda3e211d0b-metrics-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-tl898\" (UID: \"6599f4db-ad1e-4234-bf93-dcda3e211d0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898"
Nov 03 00:09:56.306007 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.306017    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6599f4db-ad1e-4234-bf93-dcda3e211d0b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-tl898\" (UID: \"6599f4db-ad1e-4234-bf93-dcda3e211d0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898"
Nov 03 00:09:56.306167 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.306040    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6599f4db-ad1e-4234-bf93-dcda3e211d0b-tmp\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-tl898\" (UID: \"6599f4db-ad1e-4234-bf93-dcda3e211d0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898"
Nov 03 00:09:56.306167 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.306094    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6599f4db-ad1e-4234-bf93-dcda3e211d0b-webhook-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-tl898\" (UID: \"6599f4db-ad1e-4234-bf93-dcda3e211d0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898"
Nov 03 00:09:56.306167 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.306163    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7khn\" (UniqueName: \"kubernetes.io/projected/6599f4db-ad1e-4234-bf93-dcda3e211d0b-kube-api-access-x7khn\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-tl898\" (UID: \"6599f4db-ad1e-4234-bf93-dcda3e211d0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898"
Nov 03 00:09:56.306347 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.306200    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6599f4db-ad1e-4234-bf93-dcda3e211d0b-manifests\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-tl898\" (UID: \"6599f4db-ad1e-4234-bf93-dcda3e211d0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898"
Nov 03 00:09:56.407082 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.407059    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6599f4db-ad1e-4234-bf93-dcda3e211d0b-manifests\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-tl898\" (UID: \"6599f4db-ad1e-4234-bf93-dcda3e211d0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898"
Nov 03 00:09:56.407200 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.407100    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6599f4db-ad1e-4234-bf93-dcda3e211d0b-metrics-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-tl898\" (UID: \"6599f4db-ad1e-4234-bf93-dcda3e211d0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898"
Nov 03 00:09:56.407200 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.407127    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6599f4db-ad1e-4234-bf93-dcda3e211d0b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-tl898\" (UID: \"6599f4db-ad1e-4234-bf93-dcda3e211d0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898"
Nov 03 00:09:56.407200 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.407149    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6599f4db-ad1e-4234-bf93-dcda3e211d0b-tmp\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-tl898\" (UID: \"6599f4db-ad1e-4234-bf93-dcda3e211d0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898"
Nov 03 00:09:56.407200 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.407170    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6599f4db-ad1e-4234-bf93-dcda3e211d0b-webhook-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-tl898\" (UID: \"6599f4db-ad1e-4234-bf93-dcda3e211d0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898"
Nov 03 00:09:56.407394 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.407205    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7khn\" (UniqueName: \"kubernetes.io/projected/6599f4db-ad1e-4234-bf93-dcda3e211d0b-kube-api-access-x7khn\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-tl898\" (UID: \"6599f4db-ad1e-4234-bf93-dcda3e211d0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898"
Nov 03 00:09:56.407531 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.407502    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6599f4db-ad1e-4234-bf93-dcda3e211d0b-manifests\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-tl898\" (UID: \"6599f4db-ad1e-4234-bf93-dcda3e211d0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898"
Nov 03 00:09:56.409869 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.409835    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6mmrg" event={"ID":"56d60aa9-02ef-45c9-be0a-37f090753315","Type":"ContainerStarted","Data":"427efdd98ba2d035edb46535aa331d29190537902a5dbc90a324e3cc8d83dd55"}
Nov 03 00:09:56.410070 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.410052    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6mmrg"
Nov 03 00:09:56.410118 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.410070    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6599f4db-ad1e-4234-bf93-dcda3e211d0b-webhook-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-tl898\" (UID: \"6599f4db-ad1e-4234-bf93-dcda3e211d0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898"
Nov 03 00:09:56.410218 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.410201    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6599f4db-ad1e-4234-bf93-dcda3e211d0b-metrics-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-tl898\" (UID: \"6599f4db-ad1e-4234-bf93-dcda3e211d0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898"
Nov 03 00:09:56.410536 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.410505    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6599f4db-ad1e-4234-bf93-dcda3e211d0b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-tl898\" (UID: \"6599f4db-ad1e-4234-bf93-dcda3e211d0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898"
Nov 03 00:09:56.411124 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.411105    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6599f4db-ad1e-4234-bf93-dcda3e211d0b-tmp\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-tl898\" (UID: \"6599f4db-ad1e-4234-bf93-dcda3e211d0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898"
Nov 03 00:09:56.424526 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.424504    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7khn\" (UniqueName: \"kubernetes.io/projected/6599f4db-ad1e-4234-bf93-dcda3e211d0b-kube-api-access-x7khn\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-tl898\" (UID: \"6599f4db-ad1e-4234-bf93-dcda3e211d0b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898"
Nov 03 00:09:56.448230 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.448189    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6mmrg" podStartSLOduration=1.4543324850000001 podStartE2EDuration="4.44817812s" totalImagesPullingTime="2.993845635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:09:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:56.446847379 +0000 UTC m=+279.326431421" watchObservedRunningTime="2026-11-03 00:09:56.44817812 +0000 UTC m=+279.327762163"
Nov 03 00:09:56.533396 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.533348    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898"
Nov 03 00:09:56.734410 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.734376    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-kbvsh"]
Nov 03 00:09:56.738881 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.738864    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kbvsh"
Nov 03 00:09:56.756078 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.756054    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-9rfhh\""
Nov 03 00:09:56.757231 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.757212    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-kbvsh"]
Nov 03 00:09:56.810995 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.810932    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhqkl\" (UniqueName: \"kubernetes.io/projected/087f9032-547d-4eec-b6fe-b6f692ef2b47-kube-api-access-rhqkl\") pod \"authorino-operator-7b7bb4bc77-kbvsh\" (UID: \"087f9032-547d-4eec-b6fe-b6f692ef2b47\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-kbvsh"
Nov 03 00:09:56.866337 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.866317    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898"]
Nov 03 00:09:56.867673 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:09:56.867646    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6599f4db_ad1e_4234_bf93_dcda3e211d0b.slice/crio-59bae11bdbd75b8741855bf9be54197c3820513716769e3d1c60d364a5f69743 WatchSource:0}: Error finding container 59bae11bdbd75b8741855bf9be54197c3820513716769e3d1c60d364a5f69743: Status 404 returned error can't find the container with id 59bae11bdbd75b8741855bf9be54197c3820513716769e3d1c60d364a5f69743
Nov 03 00:09:56.912050 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.912029    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhqkl\" (UniqueName: \"kubernetes.io/projected/087f9032-547d-4eec-b6fe-b6f692ef2b47-kube-api-access-rhqkl\") pod \"authorino-operator-7b7bb4bc77-kbvsh\" (UID: \"087f9032-547d-4eec-b6fe-b6f692ef2b47\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-kbvsh"
Nov 03 00:09:56.928863 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:56.928837    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhqkl\" (UniqueName: \"kubernetes.io/projected/087f9032-547d-4eec-b6fe-b6f692ef2b47-kube-api-access-rhqkl\") pod \"authorino-operator-7b7bb4bc77-kbvsh\" (UID: \"087f9032-547d-4eec-b6fe-b6f692ef2b47\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-kbvsh"
Nov 03 00:09:57.051765 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:57.051743    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kbvsh"
Nov 03 00:09:57.349546 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:57.349526    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-kbvsh"]
Nov 03 00:09:57.351118 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:09:57.351095    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087f9032_547d_4eec_b6fe_b6f692ef2b47.slice/crio-a8dd9784d769c59b105726298a59937131191ffe1dbc064d8f100f799e51bad4 WatchSource:0}: Error finding container a8dd9784d769c59b105726298a59937131191ffe1dbc064d8f100f799e51bad4: Status 404 returned error can't find the container with id a8dd9784d769c59b105726298a59937131191ffe1dbc064d8f100f799e51bad4
Nov 03 00:09:57.416308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:57.416278    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898" event={"ID":"6599f4db-ad1e-4234-bf93-dcda3e211d0b","Type":"ContainerStarted","Data":"59bae11bdbd75b8741855bf9be54197c3820513716769e3d1c60d364a5f69743"}
Nov 03 00:09:57.417717 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:57.417689    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kbvsh" event={"ID":"087f9032-547d-4eec-b6fe-b6f692ef2b47","Type":"ContainerStarted","Data":"a8dd9784d769c59b105726298a59937131191ffe1dbc064d8f100f799e51bad4"}
Nov 03 00:09:59.432737 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:59.432696    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kbvsh" event={"ID":"087f9032-547d-4eec-b6fe-b6f692ef2b47","Type":"ContainerStarted","Data":"8117c18e8395d48541f61a4c9a70024b7488ce04079cad91eb7a5cececf9d264"}
Nov 03 00:09:59.433221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:59.432775    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kbvsh"
Nov 03 00:09:59.468901 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:59.468848    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kbvsh" podStartSLOduration=1.840417816 podStartE2EDuration="3.468830058s" totalImagesPullingTime="1.628412242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:09:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:59.466488923 +0000 UTC m=+282.346072967" watchObservedRunningTime="2026-11-03 00:09:59.468830058 +0000 UTC m=+282.348414102"
Nov 03 00:09:59.697095 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:59.697007    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-wz9nt"]
Nov 03 00:09:59.700615 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:59.700583    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wz9nt"
Nov 03 00:09:59.713943 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:59.713921    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 03 00:09:59.714284 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:59.714259    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-tnsk4\""
Nov 03 00:09:59.726574 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:59.726538    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-wz9nt"]
Nov 03 00:09:59.840654 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:59.840618    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dxbf\" (UniqueName: \"kubernetes.io/projected/8a8544a5-55b2-4894-a924-6c408585fe18-kube-api-access-8dxbf\") pod \"dns-operator-controller-manager-8557cbdb84-wz9nt\" (UID: \"8a8544a5-55b2-4894-a924-6c408585fe18\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wz9nt"
Nov 03 00:09:59.942041 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:59.942016    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dxbf\" (UniqueName: \"kubernetes.io/projected/8a8544a5-55b2-4894-a924-6c408585fe18-kube-api-access-8dxbf\") pod \"dns-operator-controller-manager-8557cbdb84-wz9nt\" (UID: \"8a8544a5-55b2-4894-a924-6c408585fe18\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wz9nt"
Nov 03 00:09:59.967935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:09:59.967900    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dxbf\" (UniqueName: \"kubernetes.io/projected/8a8544a5-55b2-4894-a924-6c408585fe18-kube-api-access-8dxbf\") pod \"dns-operator-controller-manager-8557cbdb84-wz9nt\" (UID: \"8a8544a5-55b2-4894-a924-6c408585fe18\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wz9nt"
Nov 03 00:10:00.016952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:00.016895    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wz9nt"
Nov 03 00:10:00.386107 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:00.386080    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-wz9nt"]
Nov 03 00:10:00.387600 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:10:00.387573    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a8544a5_55b2_4894_a924_6c408585fe18.slice/crio-46772b89543a014f2034cbb8f4c1086e7c94d127587773ebccd75b4cd2791040 WatchSource:0}: Error finding container 46772b89543a014f2034cbb8f4c1086e7c94d127587773ebccd75b4cd2791040: Status 404 returned error can't find the container with id 46772b89543a014f2034cbb8f4c1086e7c94d127587773ebccd75b4cd2791040
Nov 03 00:10:00.440616 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:00.440584    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wz9nt" event={"ID":"8a8544a5-55b2-4894-a924-6c408585fe18","Type":"ContainerStarted","Data":"46772b89543a014f2034cbb8f4c1086e7c94d127587773ebccd75b4cd2791040"}
Nov 03 00:10:00.442367 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:00.442336    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898" event={"ID":"6599f4db-ad1e-4234-bf93-dcda3e211d0b","Type":"ContainerStarted","Data":"93753032dcd27653b2da04b509a3b2802a18c5e426af9ed6ffd3861d5a3f0133"}
Nov 03 00:10:01.449422 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:01.449387    5012 generic.go:356] "Generic (PLEG): container finished" podID="6599f4db-ad1e-4234-bf93-dcda3e211d0b" containerID="93753032dcd27653b2da04b509a3b2802a18c5e426af9ed6ffd3861d5a3f0133" exitCode=0
Nov 03 00:10:01.449825 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:01.449470    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898" event={"ID":"6599f4db-ad1e-4234-bf93-dcda3e211d0b","Type":"ContainerDied","Data":"93753032dcd27653b2da04b509a3b2802a18c5e426af9ed6ffd3861d5a3f0133"}
Nov 03 00:10:02.456101 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:02.456069    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898" event={"ID":"6599f4db-ad1e-4234-bf93-dcda3e211d0b","Type":"ContainerStarted","Data":"66e9b82d1a6c565a8877fc9c1b0ca853146b3ded3134812e15f3eb579a132ecc"}
Nov 03 00:10:02.456440 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:02.456203    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898"
Nov 03 00:10:02.532115 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:02.532070    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898" podStartSLOduration=2.113148464 podStartE2EDuration="6.532057151s" totalImagesPullingTime="3.917193531s" totalInitContainerRuntime="501.715156ms" isStatefulPod=false podCreationTimestamp="2026-11-03 00:09:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:02.521005757 +0000 UTC m=+285.400589800" watchObservedRunningTime="2026-11-03 00:10:02.532057151 +0000 UTC m=+285.411641194"
Nov 03 00:10:03.462968 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:03.462934    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wz9nt" event={"ID":"8a8544a5-55b2-4894-a924-6c408585fe18","Type":"ContainerStarted","Data":"d2305442074b0e55e4f9166d70eac2b23ad7f385503cbd9b9e2e127e34a1347c"}
Nov 03 00:10:03.463370 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:03.463177    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wz9nt"
Nov 03 00:10:03.514716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:03.514668    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wz9nt" podStartSLOduration=1.911744826 podStartE2EDuration="4.514654698s" totalImagesPullingTime="2.602909872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:03.51109855 +0000 UTC m=+286.390682593" watchObservedRunningTime="2026-11-03 00:10:03.514654698 +0000 UTC m=+286.394238740"
Nov 03 00:10:07.361378 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:10:07.361342    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:10:07.420828 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:07.420801    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6mmrg"
Nov 03 00:10:10.444621 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:10.444591    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-kbvsh"
Nov 03 00:10:12.875507 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:12.875477    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kvx88"]
Nov 03 00:10:12.881477 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:12.881456    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kvx88"
Nov 03 00:10:12.895193 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:12.895169    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kvx88"]
Nov 03 00:10:12.902850 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:12.902822    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:10:12.903037 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:12.902867    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 03 00:10:12.903037 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:12.902930    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-d2xm4\""
Nov 03 00:10:13.043175 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:13.043141    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bba15d9d-2067-4698-8617-c418f6f0df4b-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kvx88\" (UID: \"bba15d9d-2067-4698-8617-c418f6f0df4b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kvx88"
Nov 03 00:10:13.043323 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:13.043191    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mszs8\" (UniqueName: \"kubernetes.io/projected/bba15d9d-2067-4698-8617-c418f6f0df4b-kube-api-access-mszs8\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kvx88\" (UID: \"bba15d9d-2067-4698-8617-c418f6f0df4b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kvx88"
Nov 03 00:10:13.144103 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:13.144040    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bba15d9d-2067-4698-8617-c418f6f0df4b-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kvx88\" (UID: \"bba15d9d-2067-4698-8617-c418f6f0df4b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kvx88"
Nov 03 00:10:13.144103 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:13.144080    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mszs8\" (UniqueName: \"kubernetes.io/projected/bba15d9d-2067-4698-8617-c418f6f0df4b-kube-api-access-mszs8\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kvx88\" (UID: \"bba15d9d-2067-4698-8617-c418f6f0df4b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kvx88"
Nov 03 00:10:13.144416 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:13.144395    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bba15d9d-2067-4698-8617-c418f6f0df4b-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kvx88\" (UID: \"bba15d9d-2067-4698-8617-c418f6f0df4b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kvx88"
Nov 03 00:10:13.161141 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:13.161117    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mszs8\" (UniqueName: \"kubernetes.io/projected/bba15d9d-2067-4698-8617-c418f6f0df4b-kube-api-access-mszs8\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-kvx88\" (UID: \"bba15d9d-2067-4698-8617-c418f6f0df4b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kvx88"
Nov 03 00:10:13.194145 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:13.194122    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kvx88"
Nov 03 00:10:13.465613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:13.465545    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-tl898"
Nov 03 00:10:13.530582 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:13.530550    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kvx88"]
Nov 03 00:10:13.544062 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:10:13.544039    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba15d9d_2067_4698_8617_c418f6f0df4b.slice/crio-fb5e85b71b670a121c2b1cf84a25729a724745bcb77760198ef11f1fafeae11a WatchSource:0}: Error finding container fb5e85b71b670a121c2b1cf84a25729a724745bcb77760198ef11f1fafeae11a: Status 404 returned error can't find the container with id fb5e85b71b670a121c2b1cf84a25729a724745bcb77760198ef11f1fafeae11a
Nov 03 00:10:14.470297 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:14.470266    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wz9nt"
Nov 03 00:10:14.520331 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:14.520288    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kvx88" event={"ID":"bba15d9d-2067-4698-8617-c418f6f0df4b","Type":"ContainerStarted","Data":"fb5e85b71b670a121c2b1cf84a25729a724745bcb77760198ef11f1fafeae11a"}
Nov 03 00:10:15.527436 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:15.527366    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kvx88" event={"ID":"bba15d9d-2067-4698-8617-c418f6f0df4b","Type":"ContainerStarted","Data":"e0c4233e9fbd706944df949aceb4ad5e06f827152552090fd99bb347ab7cf1aa"}
Nov 03 00:10:15.592824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:15.592769    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-kvx88" podStartSLOduration=1.900117141 podStartE2EDuration="3.592755314s" totalImagesPullingTime="1.692638173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:15.581725051 +0000 UTC m=+298.461309094" watchObservedRunningTime="2026-11-03 00:10:15.592755314 +0000 UTC m=+298.472339356"
Nov 03 00:10:17.272708 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:17.272679    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 00:10:17.273985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:17.273959    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 00:10:17.277559 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:17.277539    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:10:17.278715 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:17.278694    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:10:20.336659 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:20.336617    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-zvstf"]
Nov 03 00:10:20.342826 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:20.342803    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zvstf"
Nov 03 00:10:20.359560 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:20.359530    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-zvstf"]
Nov 03 00:10:20.365149 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:20.365123    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 00:10:20.365261 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:20.365146    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 00:10:20.365261 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:20.365239    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-z2lgv\""
Nov 03 00:10:20.503088 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:20.503066    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtx4m\" (UniqueName: \"kubernetes.io/projected/b213fc00-129b-4ffb-a502-daa665726e79-kube-api-access-rtx4m\") pod \"cert-manager-cainjector-5c57bbd4fc-zvstf\" (UID: \"b213fc00-129b-4ffb-a502-daa665726e79\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zvstf"
Nov 03 00:10:20.503203 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:20.503100    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b213fc00-129b-4ffb-a502-daa665726e79-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-zvstf\" (UID: \"b213fc00-129b-4ffb-a502-daa665726e79\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zvstf"
Nov 03 00:10:20.604076 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:20.604051    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtx4m\" (UniqueName: \"kubernetes.io/projected/b213fc00-129b-4ffb-a502-daa665726e79-kube-api-access-rtx4m\") pod \"cert-manager-cainjector-5c57bbd4fc-zvstf\" (UID: \"b213fc00-129b-4ffb-a502-daa665726e79\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zvstf"
Nov 03 00:10:20.604210 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:20.604090    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b213fc00-129b-4ffb-a502-daa665726e79-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-zvstf\" (UID: \"b213fc00-129b-4ffb-a502-daa665726e79\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zvstf"
Nov 03 00:10:20.622538 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:20.622495    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b213fc00-129b-4ffb-a502-daa665726e79-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-zvstf\" (UID: \"b213fc00-129b-4ffb-a502-daa665726e79\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zvstf"
Nov 03 00:10:20.622971 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:20.622953    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtx4m\" (UniqueName: \"kubernetes.io/projected/b213fc00-129b-4ffb-a502-daa665726e79-kube-api-access-rtx4m\") pod \"cert-manager-cainjector-5c57bbd4fc-zvstf\" (UID: \"b213fc00-129b-4ffb-a502-daa665726e79\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zvstf"
Nov 03 00:10:20.672010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:20.671985    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zvstf"
Nov 03 00:10:20.962957 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:20.962857    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-zvstf"]
Nov 03 00:10:20.965093 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:10:20.965066    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb213fc00_129b_4ffb_a502_daa665726e79.slice/crio-d02aac134f1838eab1910479bcb2aab75e37c4545aa4dbfaf42b751d29162bb4 WatchSource:0}: Error finding container d02aac134f1838eab1910479bcb2aab75e37c4545aa4dbfaf42b751d29162bb4: Status 404 returned error can't find the container with id d02aac134f1838eab1910479bcb2aab75e37c4545aa4dbfaf42b751d29162bb4
Nov 03 00:10:21.562139 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:21.562099    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zvstf" event={"ID":"b213fc00-129b-4ffb-a502-daa665726e79","Type":"ContainerStarted","Data":"d02aac134f1838eab1910479bcb2aab75e37c4545aa4dbfaf42b751d29162bb4"}
Nov 03 00:10:24.581649 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:24.581613    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zvstf" event={"ID":"b213fc00-129b-4ffb-a502-daa665726e79","Type":"ContainerStarted","Data":"3f6d70610a9bef19ad91a5c2270bcff129c41fbda6eed4efcb527f6f61c4aee4"}
Nov 03 00:10:24.619895 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:24.619835    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zvstf" podStartSLOduration=1.4935774 podStartE2EDuration="4.619816224s" totalImagesPullingTime="3.126238824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:24.616415801 +0000 UTC m=+307.495999843" watchObservedRunningTime="2026-11-03 00:10:24.619816224 +0000 UTC m=+307.499400268"
Nov 03 00:10:32.277923 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.277836    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-66c6b6454f-8nkg7"]
Nov 03 00:10:32.333423 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.333392    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66c6b6454f-8nkg7"]
Nov 03 00:10:32.333559 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.333508    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66c6b6454f-8nkg7"
Nov 03 00:10:32.408187 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.408157    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 00:10:32.408323 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.408293    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-cwbt5\""
Nov 03 00:10:32.408627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.408614    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 00:10:32.500142 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.500109    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/851499eb-bc55-426a-be44-0538b929f027-tls-certs\") pod \"kube-auth-proxy-66c6b6454f-8nkg7\" (UID: \"851499eb-bc55-426a-be44-0538b929f027\") " pod="openshift-ingress/kube-auth-proxy-66c6b6454f-8nkg7"
Nov 03 00:10:32.500306 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.500163    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/851499eb-bc55-426a-be44-0538b929f027-tmp\") pod \"kube-auth-proxy-66c6b6454f-8nkg7\" (UID: \"851499eb-bc55-426a-be44-0538b929f027\") " pod="openshift-ingress/kube-auth-proxy-66c6b6454f-8nkg7"
Nov 03 00:10:32.500306 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.500237    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ghm6\" (UniqueName: \"kubernetes.io/projected/851499eb-bc55-426a-be44-0538b929f027-kube-api-access-4ghm6\") pod \"kube-auth-proxy-66c6b6454f-8nkg7\" (UID: \"851499eb-bc55-426a-be44-0538b929f027\") " pod="openshift-ingress/kube-auth-proxy-66c6b6454f-8nkg7"
Nov 03 00:10:32.581751 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.581678    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"]
Nov 03 00:10:32.600889 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.600862    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/851499eb-bc55-426a-be44-0538b929f027-tls-certs\") pod \"kube-auth-proxy-66c6b6454f-8nkg7\" (UID: \"851499eb-bc55-426a-be44-0538b929f027\") " pod="openshift-ingress/kube-auth-proxy-66c6b6454f-8nkg7"
Nov 03 00:10:32.601046 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.600927    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/851499eb-bc55-426a-be44-0538b929f027-tmp\") pod \"kube-auth-proxy-66c6b6454f-8nkg7\" (UID: \"851499eb-bc55-426a-be44-0538b929f027\") " pod="openshift-ingress/kube-auth-proxy-66c6b6454f-8nkg7"
Nov 03 00:10:32.601046 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.600959    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ghm6\" (UniqueName: \"kubernetes.io/projected/851499eb-bc55-426a-be44-0538b929f027-kube-api-access-4ghm6\") pod \"kube-auth-proxy-66c6b6454f-8nkg7\" (UID: \"851499eb-bc55-426a-be44-0538b929f027\") " pod="openshift-ingress/kube-auth-proxy-66c6b6454f-8nkg7"
Nov 03 00:10:32.603115 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.603093    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/851499eb-bc55-426a-be44-0538b929f027-tmp\") pod \"kube-auth-proxy-66c6b6454f-8nkg7\" (UID: \"851499eb-bc55-426a-be44-0538b929f027\") " pod="openshift-ingress/kube-auth-proxy-66c6b6454f-8nkg7"
Nov 03 00:10:32.603344 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.603324    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/851499eb-bc55-426a-be44-0538b929f027-tls-certs\") pod \"kube-auth-proxy-66c6b6454f-8nkg7\" (UID: \"851499eb-bc55-426a-be44-0538b929f027\") " pod="openshift-ingress/kube-auth-proxy-66c6b6454f-8nkg7"
Nov 03 00:10:32.610632 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.610609    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.620348 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.620329    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"]
Nov 03 00:10:32.638150 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.638124    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ghm6\" (UniqueName: \"kubernetes.io/projected/851499eb-bc55-426a-be44-0538b929f027-kube-api-access-4ghm6\") pod \"kube-auth-proxy-66c6b6454f-8nkg7\" (UID: \"851499eb-bc55-426a-be44-0538b929f027\") " pod="openshift-ingress/kube-auth-proxy-66c6b6454f-8nkg7"
Nov 03 00:10:32.664964 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.664944    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66c6b6454f-8nkg7"
Nov 03 00:10:32.679246 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.679229    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-8ztxv\""
Nov 03 00:10:32.802479 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.802448    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.802661 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.802490    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.802661 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.802532    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4z4s\" (UniqueName: \"kubernetes.io/projected/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-kube-api-access-f4z4s\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.802785 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.802669    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.802785 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.802713    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.802785 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.802773    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.802951 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.802813    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.802951 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.802844    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.802951 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.802927    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.903988 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.903951    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.904168 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.903995    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.904168 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.904021    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.904273 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.904160    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.904273 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.904195    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.904273 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.904247    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.904427 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.904301    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.904427 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.904344    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.904427 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.904396    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4z4s\" (UniqueName: \"kubernetes.io/projected/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-kube-api-access-f4z4s\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.904564 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.904432    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.904809 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.904766    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.905045 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.905021    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.905180 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.905149    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.905311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.905291    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.906931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.906889    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.907062 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.907040    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.925325 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.925296    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4z4s\" (UniqueName: \"kubernetes.io/projected/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-kube-api-access-f4z4s\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.925785 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.925760    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqchwv\" (UID: \"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.952097 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.952067    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:32.975095 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.975073    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66c6b6454f-8nkg7"]
Nov 03 00:10:32.977156 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:10:32.977131    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851499eb_bc55_426a_be44_0538b929f027.slice/crio-95c482de2efb04ee3ce08309c2aa4d8cdba62ef1b8ec0318c30846877b85d04e WatchSource:0}: Error finding container 95c482de2efb04ee3ce08309c2aa4d8cdba62ef1b8ec0318c30846877b85d04e: Status 404 returned error can't find the container with id 95c482de2efb04ee3ce08309c2aa4d8cdba62ef1b8ec0318c30846877b85d04e
Nov 03 00:10:32.980014 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:32.979996    5012 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:10:33.292556 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:33.292532    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"]
Nov 03 00:10:33.293952 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:10:33.293924    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f019f6c_5c7e_4a62_bf81_b19ea85a6d8e.slice/crio-5bf393df9fb4abc2be5bd29d84e3f66b77d10d1317d09b50b40c365ab8551e4f WatchSource:0}: Error finding container 5bf393df9fb4abc2be5bd29d84e3f66b77d10d1317d09b50b40c365ab8551e4f: Status 404 returned error can't find the container with id 5bf393df9fb4abc2be5bd29d84e3f66b77d10d1317d09b50b40c365ab8551e4f
Nov 03 00:10:33.296147 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:33.296108    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:10:33.296258 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:33.296180    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:10:33.296258 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:33.296210    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:10:33.634872 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:33.634825    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66c6b6454f-8nkg7" event={"ID":"851499eb-bc55-426a-be44-0538b929f027","Type":"ContainerStarted","Data":"95c482de2efb04ee3ce08309c2aa4d8cdba62ef1b8ec0318c30846877b85d04e"}
Nov 03 00:10:33.637005 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:33.636977    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv" event={"ID":"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e","Type":"ContainerStarted","Data":"512d8e205691b7803e5a1225c50556727141c4bd37c2d41cf5deca950f7398b0"}
Nov 03 00:10:33.637143 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:33.637028    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv" event={"ID":"6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e","Type":"ContainerStarted","Data":"5bf393df9fb4abc2be5bd29d84e3f66b77d10d1317d09b50b40c365ab8551e4f"}
Nov 03 00:10:33.705098 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:33.704460    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv" podStartSLOduration=1.704440264 podStartE2EDuration="1.704440264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:33.702111131 +0000 UTC m=+316.581695175" watchObservedRunningTime="2026-11-03 00:10:33.704440264 +0000 UTC m=+316.584024307"
Nov 03 00:10:33.953202 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:33.953124    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:34.623289 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:34.623249    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9xgct"]
Nov 03 00:10:34.633448 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:34.633419    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-9xgct"
Nov 03 00:10:34.645130 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:34.645099    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9xgct"]
Nov 03 00:10:34.664825 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:34.664802    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kqpgv\""
Nov 03 00:10:34.821699 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:34.820886    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwnl7\" (UniqueName: \"kubernetes.io/projected/ce21cfe9-84e0-4f1a-8bcb-d538e2b126a3-kube-api-access-kwnl7\") pod \"authorino-56c6c9ddd5-9xgct\" (UID: \"ce21cfe9-84e0-4f1a-8bcb-d538e2b126a3\") " pod="kuadrant-system/authorino-56c6c9ddd5-9xgct"
Nov 03 00:10:34.921846 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:34.921764    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwnl7\" (UniqueName: \"kubernetes.io/projected/ce21cfe9-84e0-4f1a-8bcb-d538e2b126a3-kube-api-access-kwnl7\") pod \"authorino-56c6c9ddd5-9xgct\" (UID: \"ce21cfe9-84e0-4f1a-8bcb-d538e2b126a3\") " pod="kuadrant-system/authorino-56c6c9ddd5-9xgct"
Nov 03 00:10:34.939105 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:34.939078    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwnl7\" (UniqueName: \"kubernetes.io/projected/ce21cfe9-84e0-4f1a-8bcb-d538e2b126a3-kube-api-access-kwnl7\") pod \"authorino-56c6c9ddd5-9xgct\" (UID: \"ce21cfe9-84e0-4f1a-8bcb-d538e2b126a3\") " pod="kuadrant-system/authorino-56c6c9ddd5-9xgct"
Nov 03 00:10:34.957452 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:34.957426    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-9xgct"
Nov 03 00:10:34.958041 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:34.958018    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:35.258787 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:35.258709    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9xgct"]
Nov 03 00:10:35.263120 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:10:35.263092    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce21cfe9_84e0_4f1a_8bcb_d538e2b126a3.slice/crio-bc0b3c7a5332806ea90e1d8d58508cf1b7cb577330fb171d41f382a21ad59bac WatchSource:0}: Error finding container bc0b3c7a5332806ea90e1d8d58508cf1b7cb577330fb171d41f382a21ad59bac: Status 404 returned error can't find the container with id bc0b3c7a5332806ea90e1d8d58508cf1b7cb577330fb171d41f382a21ad59bac
Nov 03 00:10:35.651308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:35.651275    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66c6b6454f-8nkg7" event={"ID":"851499eb-bc55-426a-be44-0538b929f027","Type":"ContainerStarted","Data":"9d4c41f649c85a1589591e01a03a9a4d22bbae22189b342152e7194370316cc2"}
Nov 03 00:10:35.652587 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:35.652562    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-9xgct" event={"ID":"ce21cfe9-84e0-4f1a-8bcb-d538e2b126a3","Type":"ContainerStarted","Data":"bc0b3c7a5332806ea90e1d8d58508cf1b7cb577330fb171d41f382a21ad59bac"}
Nov 03 00:10:35.652786 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:35.652776    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:35.653633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:35.653613    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqchwv"
Nov 03 00:10:35.687433 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:35.687384    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-66c6b6454f-8nkg7" podStartSLOduration=1.3136424 podStartE2EDuration="3.687371024s" totalImagesPullingTime="2.373728624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:35.685108168 +0000 UTC m=+318.564692212" watchObservedRunningTime="2026-11-03 00:10:35.687371024 +0000 UTC m=+318.566955066"
Nov 03 00:10:37.361628 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:10:37.361596    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:10:37.666415 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:37.666333    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-9xgct" event={"ID":"ce21cfe9-84e0-4f1a-8bcb-d538e2b126a3","Type":"ContainerStarted","Data":"e05e0ab1108eccc28723283edb860858196ec5c4f2ef552ee5cce192600a0f41"}
Nov 03 00:10:37.701022 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:10:37.700975    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-9xgct" podStartSLOduration=1.561537903 podStartE2EDuration="3.700962645s" totalImagesPullingTime="2.139424742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:37.699366661 +0000 UTC m=+320.578950707" watchObservedRunningTime="2026-11-03 00:10:37.700962645 +0000 UTC m=+320.580546687"
Nov 03 00:11:07.362560 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:11:07.362522    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:11:37.358874 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:11:37.358842    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-x5z9p"]
Nov 03 00:11:37.362300 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:11:37.362259    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:11:37.362689 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:11:37.362660    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-x5z9p"
Nov 03 00:11:37.376252 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:11:37.376229    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 00:11:37.386327 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:11:37.386300    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 00:11:37.386466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:11:37.386351    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 03 00:11:37.386466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:11:37.386373    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 03 00:11:37.386466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:11:37.386411    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2z8mt\""
Nov 03 00:11:37.390152 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:11:37.390128    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-x5z9p"]
Nov 03 00:11:37.507199 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:11:37.507171    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgchd\" (UniqueName: \"kubernetes.io/projected/8c5d5029-6d11-48ee-8771-d2bb8cdcde13-kube-api-access-hgchd\") pod \"odh-model-controller-769875db56-x5z9p\" (UID: \"8c5d5029-6d11-48ee-8771-d2bb8cdcde13\") " pod="opendatahub/odh-model-controller-769875db56-x5z9p"
Nov 03 00:11:37.507320 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:11:37.507233    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c5d5029-6d11-48ee-8771-d2bb8cdcde13-metrics-certs\") pod \"odh-model-controller-769875db56-x5z9p\" (UID: \"8c5d5029-6d11-48ee-8771-d2bb8cdcde13\") " pod="opendatahub/odh-model-controller-769875db56-x5z9p"
Nov 03 00:11:37.507386 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:11:37.507357    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c5d5029-6d11-48ee-8771-d2bb8cdcde13-cert\") pod \"odh-model-controller-769875db56-x5z9p\" (UID: \"8c5d5029-6d11-48ee-8771-d2bb8cdcde13\") " pod="opendatahub/odh-model-controller-769875db56-x5z9p"
Nov 03 00:11:37.608623 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:11:37.608598    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c5d5029-6d11-48ee-8771-d2bb8cdcde13-metrics-certs\") pod \"odh-model-controller-769875db56-x5z9p\" (UID: \"8c5d5029-6d11-48ee-8771-d2bb8cdcde13\") " pod="opendatahub/odh-model-controller-769875db56-x5z9p"
Nov 03 00:11:37.608732 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:11:37.608639    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c5d5029-6d11-48ee-8771-d2bb8cdcde13-cert\") pod \"odh-model-controller-769875db56-x5z9p\" (UID: \"8c5d5029-6d11-48ee-8771-d2bb8cdcde13\") " pod="opendatahub/odh-model-controller-769875db56-x5z9p"
Nov 03 00:11:37.608732 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:11:37.608669    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgchd\" (UniqueName: \"kubernetes.io/projected/8c5d5029-6d11-48ee-8771-d2bb8cdcde13-kube-api-access-hgchd\") pod \"odh-model-controller-769875db56-x5z9p\" (UID: \"8c5d5029-6d11-48ee-8771-d2bb8cdcde13\") " pod="opendatahub/odh-model-controller-769875db56-x5z9p"
Nov 03 00:11:37.611078 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:11:37.611028    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c5d5029-6d11-48ee-8771-d2bb8cdcde13-metrics-certs\") pod \"odh-model-controller-769875db56-x5z9p\" (UID: \"8c5d5029-6d11-48ee-8771-d2bb8cdcde13\") " pod="opendatahub/odh-model-controller-769875db56-x5z9p"
Nov 03 00:11:37.611172 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:11:37.611154    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c5d5029-6d11-48ee-8771-d2bb8cdcde13-cert\") pod \"odh-model-controller-769875db56-x5z9p\" (UID: \"8c5d5029-6d11-48ee-8771-d2bb8cdcde13\") " pod="opendatahub/odh-model-controller-769875db56-x5z9p"
Nov 03 00:11:37.625320 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:11:37.625297    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgchd\" (UniqueName: \"kubernetes.io/projected/8c5d5029-6d11-48ee-8771-d2bb8cdcde13-kube-api-access-hgchd\") pod \"odh-model-controller-769875db56-x5z9p\" (UID: \"8c5d5029-6d11-48ee-8771-d2bb8cdcde13\") " pod="opendatahub/odh-model-controller-769875db56-x5z9p"
Nov 03 00:11:37.679359 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:11:37.679334    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-x5z9p"
Nov 03 00:11:37.972878 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:11:37.972852    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-x5z9p"]
Nov 03 00:11:37.973897 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:11:37.973870    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5d5029_6d11_48ee_8771_d2bb8cdcde13.slice/crio-360365b32b65ebe78f95d06e80e80165d49d16d41c3c87b1792a697be8f25cd9 WatchSource:0}: Error finding container 360365b32b65ebe78f95d06e80e80165d49d16d41c3c87b1792a697be8f25cd9: Status 404 returned error can't find the container with id 360365b32b65ebe78f95d06e80e80165d49d16d41c3c87b1792a697be8f25cd9
Nov 03 00:11:38.007672 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:11:38.007639    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-x5z9p" event={"ID":"8c5d5029-6d11-48ee-8771-d2bb8cdcde13","Type":"ContainerStarted","Data":"360365b32b65ebe78f95d06e80e80165d49d16d41c3c87b1792a697be8f25cd9"}
Nov 03 00:11:41.029758 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:11:41.029676    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-x5z9p" event={"ID":"8c5d5029-6d11-48ee-8771-d2bb8cdcde13","Type":"ContainerStarted","Data":"4fac52b36734d8b1c50e4f01fd13e0c51c327e3344b263041e8ef9c36db1d9ce"}
Nov 03 00:11:41.030153 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:11:41.029859    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-x5z9p"
Nov 03 00:11:41.063238 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:11:41.063183    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-x5z9p" podStartSLOduration=1.415675006 podStartE2EDuration="4.063169345s" totalImagesPullingTime="2.647494339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:11:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:11:41.062838796 +0000 UTC m=+383.942422839" watchObservedRunningTime="2026-11-03 00:11:41.063169345 +0000 UTC m=+383.942753387"
Nov 03 00:11:52.038487 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:11:52.038453    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-x5z9p"
Nov 03 00:12:07.361249 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:12:07.361172    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:12:11.704416 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:11.704334    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-8fzrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Nov 03 00:12:11.704416 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:11.704397    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc" podUID="51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Nov 03 00:12:11.878593 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:11.878563    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hzflm"]
Nov 03 00:12:11.882121 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:11.882100    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hzflm"
Nov 03 00:12:11.896354 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:11.896332    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hzflm"]
Nov 03 00:12:11.972989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:11.972906    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnhdn\" (UniqueName: \"kubernetes.io/projected/f911c626-7e8e-47ff-80b7-2f485a3ebb4d-kube-api-access-nnhdn\") pod \"authorino-6f6944bcdd-hzflm\" (UID: \"f911c626-7e8e-47ff-80b7-2f485a3ebb4d\") " pod="kuadrant-system/authorino-6f6944bcdd-hzflm"
Nov 03 00:12:12.073863 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.073833    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnhdn\" (UniqueName: \"kubernetes.io/projected/f911c626-7e8e-47ff-80b7-2f485a3ebb4d-kube-api-access-nnhdn\") pod \"authorino-6f6944bcdd-hzflm\" (UID: \"f911c626-7e8e-47ff-80b7-2f485a3ebb4d\") " pod="kuadrant-system/authorino-6f6944bcdd-hzflm"
Nov 03 00:12:12.093563 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.093540    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnhdn\" (UniqueName: \"kubernetes.io/projected/f911c626-7e8e-47ff-80b7-2f485a3ebb4d-kube-api-access-nnhdn\") pod \"authorino-6f6944bcdd-hzflm\" (UID: \"f911c626-7e8e-47ff-80b7-2f485a3ebb4d\") " pod="kuadrant-system/authorino-6f6944bcdd-hzflm"
Nov 03 00:12:12.177837 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.177810    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hzflm"]
Nov 03 00:12:12.178008 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.177996    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hzflm"
Nov 03 00:12:12.199759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.199734    5012 generic.go:356] "Generic (PLEG): container finished" podID="51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" containerID="18a8893cac47b63fca1c159aea7507e8c7a080459cbadd0005de6e25326804d6" exitCode=0
Nov 03 00:12:12.199870 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.199804    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc" event={"ID":"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c","Type":"ContainerDied","Data":"18a8893cac47b63fca1c159aea7507e8c7a080459cbadd0005de6e25326804d6"}
Nov 03 00:12:12.200217 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.200198    5012 scope.go:122] "RemoveContainer" containerID="18a8893cac47b63fca1c159aea7507e8c7a080459cbadd0005de6e25326804d6"
Nov 03 00:12:12.201554 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.201521    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:12:12.201671 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.201605    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:12:12.201671 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.201647    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:12:12.203473 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.203453    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-65g4t"]
Nov 03 00:12:12.214157 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.214132    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-65g4t"
Nov 03 00:12:12.224162 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.224102    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-65g4t"]
Nov 03 00:12:12.227196 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.227178    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 03 00:12:12.266559 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.266529    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-65g4t"]
Nov 03 00:12:12.266918 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:12:12.266886    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-vpwkj tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-vpwkj tls-cert]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-65g4t" podUID="7b069ee7-bc61-479a-aafb-1c72b1cecff9"
Nov 03 00:12:12.277384 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.277356    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7b069ee7-bc61-479a-aafb-1c72b1cecff9-tls-cert\") pod \"authorino-6889cbd8c4-65g4t\" (UID: \"7b069ee7-bc61-479a-aafb-1c72b1cecff9\") " pod="kuadrant-system/authorino-6889cbd8c4-65g4t"
Nov 03 00:12:12.277469 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.277399    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpwkj\" (UniqueName: \"kubernetes.io/projected/7b069ee7-bc61-479a-aafb-1c72b1cecff9-kube-api-access-vpwkj\") pod \"authorino-6889cbd8c4-65g4t\" (UID: \"7b069ee7-bc61-479a-aafb-1c72b1cecff9\") " pod="kuadrant-system/authorino-6889cbd8c4-65g4t"
Nov 03 00:12:12.291517 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.291481    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56bf98447d-nrnvg"]
Nov 03 00:12:12.297046 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.294994    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56bf98447d-nrnvg"
Nov 03 00:12:12.314674 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.312171    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56bf98447d-nrnvg"]
Nov 03 00:12:12.377824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.377790    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkd9k\" (UniqueName: \"kubernetes.io/projected/7a662a32-1c38-4a1a-a49a-b2b90940582e-kube-api-access-fkd9k\") pod \"authorino-56bf98447d-nrnvg\" (UID: \"7a662a32-1c38-4a1a-a49a-b2b90940582e\") " pod="kuadrant-system/authorino-56bf98447d-nrnvg"
Nov 03 00:12:12.378036 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.377835    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7b069ee7-bc61-479a-aafb-1c72b1cecff9-tls-cert\") pod \"authorino-6889cbd8c4-65g4t\" (UID: \"7b069ee7-bc61-479a-aafb-1c72b1cecff9\") " pod="kuadrant-system/authorino-6889cbd8c4-65g4t"
Nov 03 00:12:12.378036 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.377855    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpwkj\" (UniqueName: \"kubernetes.io/projected/7b069ee7-bc61-479a-aafb-1c72b1cecff9-kube-api-access-vpwkj\") pod \"authorino-6889cbd8c4-65g4t\" (UID: \"7b069ee7-bc61-479a-aafb-1c72b1cecff9\") " pod="kuadrant-system/authorino-6889cbd8c4-65g4t"
Nov 03 00:12:12.378036 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.377990    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7a662a32-1c38-4a1a-a49a-b2b90940582e-tls-cert\") pod \"authorino-56bf98447d-nrnvg\" (UID: \"7a662a32-1c38-4a1a-a49a-b2b90940582e\") " pod="kuadrant-system/authorino-56bf98447d-nrnvg"
Nov 03 00:12:12.380403 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.380373    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7b069ee7-bc61-479a-aafb-1c72b1cecff9-tls-cert\") pod \"authorino-6889cbd8c4-65g4t\" (UID: \"7b069ee7-bc61-479a-aafb-1c72b1cecff9\") " pod="kuadrant-system/authorino-6889cbd8c4-65g4t"
Nov 03 00:12:12.394761 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.394734    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpwkj\" (UniqueName: \"kubernetes.io/projected/7b069ee7-bc61-479a-aafb-1c72b1cecff9-kube-api-access-vpwkj\") pod \"authorino-6889cbd8c4-65g4t\" (UID: \"7b069ee7-bc61-479a-aafb-1c72b1cecff9\") " pod="kuadrant-system/authorino-6889cbd8c4-65g4t"
Nov 03 00:12:12.475844 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.475821    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hzflm"]
Nov 03 00:12:12.478145 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:12:12.478117    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf911c626_7e8e_47ff_80b7_2f485a3ebb4d.slice/crio-1d12922ce11d627e3fb6c591b6a79820e30e951ea4e5ad5122c53205a2fb6912 WatchSource:0}: Error finding container 1d12922ce11d627e3fb6c591b6a79820e30e951ea4e5ad5122c53205a2fb6912: Status 404 returned error can't find the container with id 1d12922ce11d627e3fb6c591b6a79820e30e951ea4e5ad5122c53205a2fb6912
Nov 03 00:12:12.478468 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.478452    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7a662a32-1c38-4a1a-a49a-b2b90940582e-tls-cert\") pod \"authorino-56bf98447d-nrnvg\" (UID: \"7a662a32-1c38-4a1a-a49a-b2b90940582e\") " pod="kuadrant-system/authorino-56bf98447d-nrnvg"
Nov 03 00:12:12.478536 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.478509    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkd9k\" (UniqueName: \"kubernetes.io/projected/7a662a32-1c38-4a1a-a49a-b2b90940582e-kube-api-access-fkd9k\") pod \"authorino-56bf98447d-nrnvg\" (UID: \"7a662a32-1c38-4a1a-a49a-b2b90940582e\") " pod="kuadrant-system/authorino-56bf98447d-nrnvg"
Nov 03 00:12:12.481259 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.481237    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7a662a32-1c38-4a1a-a49a-b2b90940582e-tls-cert\") pod \"authorino-56bf98447d-nrnvg\" (UID: \"7a662a32-1c38-4a1a-a49a-b2b90940582e\") " pod="kuadrant-system/authorino-56bf98447d-nrnvg"
Nov 03 00:12:12.494780 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.494756    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkd9k\" (UniqueName: \"kubernetes.io/projected/7a662a32-1c38-4a1a-a49a-b2b90940582e-kube-api-access-fkd9k\") pod \"authorino-56bf98447d-nrnvg\" (UID: \"7a662a32-1c38-4a1a-a49a-b2b90940582e\") " pod="kuadrant-system/authorino-56bf98447d-nrnvg"
Nov 03 00:12:12.609143 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.609084    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56bf98447d-nrnvg"
Nov 03 00:12:12.912158 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:12.911339    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56bf98447d-nrnvg"]
Nov 03 00:12:12.914773 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:12:12.914741    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a662a32_1c38_4a1a_a49a_b2b90940582e.slice/crio-c7dda050089e2d0745fe59abf3a960f853e82bc4c3a2c52ed383787ffcd91065 WatchSource:0}: Error finding container c7dda050089e2d0745fe59abf3a960f853e82bc4c3a2c52ed383787ffcd91065: Status 404 returned error can't find the container with id c7dda050089e2d0745fe59abf3a960f853e82bc4c3a2c52ed383787ffcd91065
Nov 03 00:12:13.216288 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:13.216190    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56bf98447d-nrnvg" event={"ID":"7a662a32-1c38-4a1a-a49a-b2b90940582e","Type":"ContainerStarted","Data":"c7dda050089e2d0745fe59abf3a960f853e82bc4c3a2c52ed383787ffcd91065"}
Nov 03 00:12:13.218259 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:13.218228    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc" event={"ID":"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c","Type":"ContainerStarted","Data":"5439684e89b69f602bda859029a342e8564bf1754cf650752edcf7eebe8f9730"}
Nov 03 00:12:13.219661 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:13.219641    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-65g4t"
Nov 03 00:12:13.219738 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:13.219634    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-hzflm" event={"ID":"f911c626-7e8e-47ff-80b7-2f485a3ebb4d","Type":"ContainerStarted","Data":"1d12922ce11d627e3fb6c591b6a79820e30e951ea4e5ad5122c53205a2fb6912"}
Nov 03 00:12:13.228013 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:13.227992    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-65g4t"
Nov 03 00:12:13.413102 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:13.413065    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b069ee7-bc61-479a-aafb-1c72b1cecff9-kube-api-access-vpwkj\" (UniqueName: \"kubernetes.io/projected/7b069ee7-bc61-479a-aafb-1c72b1cecff9-kube-api-access-vpwkj\") pod \"7b069ee7-bc61-479a-aafb-1c72b1cecff9\" (UID: \"7b069ee7-bc61-479a-aafb-1c72b1cecff9\") "
Nov 03 00:12:13.413102 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:13.413107    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b069ee7-bc61-479a-aafb-1c72b1cecff9-tls-cert\" (UniqueName: \"kubernetes.io/secret/7b069ee7-bc61-479a-aafb-1c72b1cecff9-tls-cert\") pod \"7b069ee7-bc61-479a-aafb-1c72b1cecff9\" (UID: \"7b069ee7-bc61-479a-aafb-1c72b1cecff9\") "
Nov 03 00:12:13.415341 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:13.415313    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b069ee7-bc61-479a-aafb-1c72b1cecff9-tls-cert" pod "7b069ee7-bc61-479a-aafb-1c72b1cecff9" (UID: "7b069ee7-bc61-479a-aafb-1c72b1cecff9"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:12:13.415447 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:13.415337    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b069ee7-bc61-479a-aafb-1c72b1cecff9-kube-api-access-vpwkj" pod "7b069ee7-bc61-479a-aafb-1c72b1cecff9" (UID: "7b069ee7-bc61-479a-aafb-1c72b1cecff9"). InnerVolumeSpecName "kube-api-access-vpwkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:12:13.514290 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:13.514252    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpwkj\" (UniqueName: \"kubernetes.io/projected/7b069ee7-bc61-479a-aafb-1c72b1cecff9-kube-api-access-vpwkj\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:12:13.514290 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:13.514288    5012 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7b069ee7-bc61-479a-aafb-1c72b1cecff9-tls-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:12:13.689937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:13.689423    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:12:13.689937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:13.689553    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-8fzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Nov 03 00:12:13.689937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:13.689600    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc" podUID="51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Nov 03 00:12:14.233480 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:14.231747    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56bf98447d-nrnvg" event={"ID":"7a662a32-1c38-4a1a-a49a-b2b90940582e","Type":"ContainerStarted","Data":"6606bfe28e51df645a82b400ac338de44320302dc4be6099c957cc18a52ecf31"}
Nov 03 00:12:14.234007 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:14.233746    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-hzflm" event={"ID":"f911c626-7e8e-47ff-80b7-2f485a3ebb4d","Type":"ContainerStarted","Data":"538fdcffb1cbb225a9e26c41a0e5450129e2ea66c2000c300ef1683f10047cc8"}
Nov 03 00:12:14.234007 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:14.233780    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-hzflm" podUID="f911c626-7e8e-47ff-80b7-2f485a3ebb4d" containerName="authorino" containerID="cri-o://538fdcffb1cbb225a9e26c41a0e5450129e2ea66c2000c300ef1683f10047cc8" gracePeriod=30
Nov 03 00:12:14.234131 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:14.234112    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-65g4t"
Nov 03 00:12:14.269057 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:14.269006    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56bf98447d-nrnvg" podStartSLOduration=1.648960041 podStartE2EDuration="2.268992111s" totalImagesPullingTime="620.03207ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:12:14.265252797 +0000 UTC m=+417.144836841" watchObservedRunningTime="2026-11-03 00:12:14.268992111 +0000 UTC m=+417.148576154"
Nov 03 00:12:14.324511 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:14.319053    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9xgct"]
Nov 03 00:12:14.324511 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:14.319284    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-9xgct" podUID="ce21cfe9-84e0-4f1a-8bcb-d538e2b126a3" containerName="authorino" containerID="cri-o://e05e0ab1108eccc28723283edb860858196ec5c4f2ef552ee5cce192600a0f41" gracePeriod=30
Nov 03 00:12:14.332436 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:14.327767    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-65g4t"]
Nov 03 00:12:14.332436 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:14.331208    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-65g4t"]
Nov 03 00:12:14.359108 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:14.359052    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-hzflm" podStartSLOduration=2.565520905 podStartE2EDuration="3.359037884s" totalImagesPullingTime="793.516979ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:12:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:12:14.357609091 +0000 UTC m=+417.237193135" watchObservedRunningTime="2026-11-03 00:12:14.359037884 +0000 UTC m=+417.238621924"
Nov 03 00:12:14.562252 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:14.562225    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hzflm"
Nov 03 00:12:14.629965 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:14.629898    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f911c626-7e8e-47ff-80b7-2f485a3ebb4d-kube-api-access-nnhdn\" (UniqueName: \"kubernetes.io/projected/f911c626-7e8e-47ff-80b7-2f485a3ebb4d-kube-api-access-nnhdn\") pod \"f911c626-7e8e-47ff-80b7-2f485a3ebb4d\" (UID: \"f911c626-7e8e-47ff-80b7-2f485a3ebb4d\") "
Nov 03 00:12:14.632376 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:14.632347    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f911c626-7e8e-47ff-80b7-2f485a3ebb4d-kube-api-access-nnhdn" pod "f911c626-7e8e-47ff-80b7-2f485a3ebb4d" (UID: "f911c626-7e8e-47ff-80b7-2f485a3ebb4d"). InnerVolumeSpecName "kube-api-access-nnhdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:12:14.651759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:14.651733    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-9xgct"
Nov 03 00:12:14.711066 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:14.711024    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-8fzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Nov 03 00:12:14.711232 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:14.711086    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc" podUID="51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Nov 03 00:12:14.734104 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:14.734070    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce21cfe9-84e0-4f1a-8bcb-d538e2b126a3-kube-api-access-kwnl7\" (UniqueName: \"kubernetes.io/projected/ce21cfe9-84e0-4f1a-8bcb-d538e2b126a3-kube-api-access-kwnl7\") pod \"ce21cfe9-84e0-4f1a-8bcb-d538e2b126a3\" (UID: \"ce21cfe9-84e0-4f1a-8bcb-d538e2b126a3\") "
Nov 03 00:12:14.734375 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:14.734349    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnhdn\" (UniqueName: \"kubernetes.io/projected/f911c626-7e8e-47ff-80b7-2f485a3ebb4d-kube-api-access-nnhdn\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:12:14.736206 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:14.736173    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce21cfe9-84e0-4f1a-8bcb-d538e2b126a3-kube-api-access-kwnl7" pod "ce21cfe9-84e0-4f1a-8bcb-d538e2b126a3" (UID: "ce21cfe9-84e0-4f1a-8bcb-d538e2b126a3"). InnerVolumeSpecName "kube-api-access-kwnl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:12:14.835709 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:14.835639    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwnl7\" (UniqueName: \"kubernetes.io/projected/ce21cfe9-84e0-4f1a-8bcb-d538e2b126a3-kube-api-access-kwnl7\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:12:15.240900 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:15.240867    5012 generic.go:356] "Generic (PLEG): container finished" podID="ce21cfe9-84e0-4f1a-8bcb-d538e2b126a3" containerID="e05e0ab1108eccc28723283edb860858196ec5c4f2ef552ee5cce192600a0f41" exitCode=0
Nov 03 00:12:15.241341 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:15.241006    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-9xgct"
Nov 03 00:12:15.242027 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:15.241996    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b069ee7-bc61-479a-aafb-1c72b1cecff9" path="/var/lib/kubelet/pods/7b069ee7-bc61-479a-aafb-1c72b1cecff9/volumes"
Nov 03 00:12:15.242271 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:15.242252    5012 generic.go:356] "Generic (PLEG): container finished" podID="f911c626-7e8e-47ff-80b7-2f485a3ebb4d" containerID="538fdcffb1cbb225a9e26c41a0e5450129e2ea66c2000c300ef1683f10047cc8" exitCode=0
Nov 03 00:12:15.242368 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:15.242355    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hzflm"
Nov 03 00:12:15.242443 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:15.242422    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-9xgct" event={"ID":"ce21cfe9-84e0-4f1a-8bcb-d538e2b126a3","Type":"ContainerDied","Data":"e05e0ab1108eccc28723283edb860858196ec5c4f2ef552ee5cce192600a0f41"}
Nov 03 00:12:15.242497 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:15.242462    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-9xgct" event={"ID":"ce21cfe9-84e0-4f1a-8bcb-d538e2b126a3","Type":"ContainerDied","Data":"bc0b3c7a5332806ea90e1d8d58508cf1b7cb577330fb171d41f382a21ad59bac"}
Nov 03 00:12:15.242497 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:15.242482    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-hzflm" event={"ID":"f911c626-7e8e-47ff-80b7-2f485a3ebb4d","Type":"ContainerDied","Data":"538fdcffb1cbb225a9e26c41a0e5450129e2ea66c2000c300ef1683f10047cc8"}
Nov 03 00:12:15.242604 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:15.242500    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-hzflm" event={"ID":"f911c626-7e8e-47ff-80b7-2f485a3ebb4d","Type":"ContainerDied","Data":"1d12922ce11d627e3fb6c591b6a79820e30e951ea4e5ad5122c53205a2fb6912"}
Nov 03 00:12:15.242604 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:15.242530    5012 scope.go:122] "RemoveContainer" containerID="e05e0ab1108eccc28723283edb860858196ec5c4f2ef552ee5cce192600a0f41"
Nov 03 00:12:15.253439 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:15.253423    5012 scope.go:122] "RemoveContainer" containerID="e05e0ab1108eccc28723283edb860858196ec5c4f2ef552ee5cce192600a0f41"
Nov 03 00:12:15.253769 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:12:15.253734    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e05e0ab1108eccc28723283edb860858196ec5c4f2ef552ee5cce192600a0f41\": container with ID starting with e05e0ab1108eccc28723283edb860858196ec5c4f2ef552ee5cce192600a0f41 not found: ID does not exist" containerID="e05e0ab1108eccc28723283edb860858196ec5c4f2ef552ee5cce192600a0f41"
Nov 03 00:12:15.253873 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:15.253769    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05e0ab1108eccc28723283edb860858196ec5c4f2ef552ee5cce192600a0f41"} err="failed to get container status \"e05e0ab1108eccc28723283edb860858196ec5c4f2ef552ee5cce192600a0f41\": rpc error: code = NotFound desc = could not find container \"e05e0ab1108eccc28723283edb860858196ec5c4f2ef552ee5cce192600a0f41\": container with ID starting with e05e0ab1108eccc28723283edb860858196ec5c4f2ef552ee5cce192600a0f41 not found: ID does not exist"
Nov 03 00:12:15.253873 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:15.253795    5012 scope.go:122] "RemoveContainer" containerID="538fdcffb1cbb225a9e26c41a0e5450129e2ea66c2000c300ef1683f10047cc8"
Nov 03 00:12:15.261761 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:15.261745    5012 scope.go:122] "RemoveContainer" containerID="538fdcffb1cbb225a9e26c41a0e5450129e2ea66c2000c300ef1683f10047cc8"
Nov 03 00:12:15.262029 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:12:15.262006    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"538fdcffb1cbb225a9e26c41a0e5450129e2ea66c2000c300ef1683f10047cc8\": container with ID starting with 538fdcffb1cbb225a9e26c41a0e5450129e2ea66c2000c300ef1683f10047cc8 not found: ID does not exist" containerID="538fdcffb1cbb225a9e26c41a0e5450129e2ea66c2000c300ef1683f10047cc8"
Nov 03 00:12:15.262098 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:15.262039    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538fdcffb1cbb225a9e26c41a0e5450129e2ea66c2000c300ef1683f10047cc8"} err="failed to get container status \"538fdcffb1cbb225a9e26c41a0e5450129e2ea66c2000c300ef1683f10047cc8\": rpc error: code = NotFound desc = could not find container \"538fdcffb1cbb225a9e26c41a0e5450129e2ea66c2000c300ef1683f10047cc8\": container with ID starting with 538fdcffb1cbb225a9e26c41a0e5450129e2ea66c2000c300ef1683f10047cc8 not found: ID does not exist"
Nov 03 00:12:15.313592 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:15.313565    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hzflm"]
Nov 03 00:12:15.315635 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:15.315615    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hzflm"]
Nov 03 00:12:15.341109 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:15.341087    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9xgct"]
Nov 03 00:12:15.343040 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:15.343019    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9xgct"]
Nov 03 00:12:15.702743 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:15.702293    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-8fzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Nov 03 00:12:15.702743 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:15.702354    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc" podUID="51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Nov 03 00:12:16.701933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:16.699264    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-8fzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Nov 03 00:12:16.701933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:16.699339    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc" podUID="51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Nov 03 00:12:17.242385 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:17.242349    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce21cfe9-84e0-4f1a-8bcb-d538e2b126a3" path="/var/lib/kubelet/pods/ce21cfe9-84e0-4f1a-8bcb-d538e2b126a3/volumes"
Nov 03 00:12:17.242679 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:17.242665    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f911c626-7e8e-47ff-80b7-2f485a3ebb4d" path="/var/lib/kubelet/pods/f911c626-7e8e-47ff-80b7-2f485a3ebb4d/volumes"
Nov 03 00:12:17.701932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:17.701248    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-8fzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Nov 03 00:12:17.701932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:17.701305    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc" podUID="51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Nov 03 00:12:18.692887 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:18.692852    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:12:18.693213 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:18.693189    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:12:18.693887 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:12:18.693869    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:12:37.363433 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:12:37.363395    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:13:07.362071 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:13:07.362033    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:13:13.221950 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:13.221900    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-885tl"]
Nov 03 00:13:13.222525 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:13.222460    5012 cpu_manager.go:414] "containerMap: removing container" podUID="f911c626-7e8e-47ff-80b7-2f485a3ebb4d" containerName="authorino"
Nov 03 00:13:13.222525 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:13.222490    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f911c626-7e8e-47ff-80b7-2f485a3ebb4d" containerName="authorino"
Nov 03 00:13:13.222525 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:13.222519    5012 cpu_manager.go:414] "containerMap: removing container" podUID="ce21cfe9-84e0-4f1a-8bcb-d538e2b126a3" containerName="authorino"
Nov 03 00:13:13.222700 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:13.222527    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce21cfe9-84e0-4f1a-8bcb-d538e2b126a3" containerName="authorino"
Nov 03 00:13:13.222700 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:13.222657    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce21cfe9-84e0-4f1a-8bcb-d538e2b126a3" containerName="authorino"
Nov 03 00:13:13.222700 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:13.222678    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="f911c626-7e8e-47ff-80b7-2f485a3ebb4d" containerName="authorino"
Nov 03 00:13:13.225716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:13.225694    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-885tl"
Nov 03 00:13:13.243527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:13.243506    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-885tl"]
Nov 03 00:13:13.250154 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:13.250137    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-csh4z\""
Nov 03 00:13:13.308695 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:13.308665    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 03 00:13:13.378419 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:13.378376    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df410080-3e08-4f8d-b1d3-84db851d8782-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-885tl\" (UID: \"df410080-3e08-4f8d-b1d3-84db851d8782\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-885tl"
Nov 03 00:13:13.378607 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:13.378550    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l87z\" (UniqueName: \"kubernetes.io/projected/df410080-3e08-4f8d-b1d3-84db851d8782-kube-api-access-8l87z\") pod \"payload-processing-55fbbbf9dc-885tl\" (UID: \"df410080-3e08-4f8d-b1d3-84db851d8782\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-885tl"
Nov 03 00:13:13.479478 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:13.479384    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l87z\" (UniqueName: \"kubernetes.io/projected/df410080-3e08-4f8d-b1d3-84db851d8782-kube-api-access-8l87z\") pod \"payload-processing-55fbbbf9dc-885tl\" (UID: \"df410080-3e08-4f8d-b1d3-84db851d8782\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-885tl"
Nov 03 00:13:13.479478 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:13.479447    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df410080-3e08-4f8d-b1d3-84db851d8782-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-885tl\" (UID: \"df410080-3e08-4f8d-b1d3-84db851d8782\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-885tl"
Nov 03 00:13:13.480396 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:13.480368    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df410080-3e08-4f8d-b1d3-84db851d8782-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-885tl\" (UID: \"df410080-3e08-4f8d-b1d3-84db851d8782\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-885tl"
Nov 03 00:13:13.496098 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:13.496075    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l87z\" (UniqueName: \"kubernetes.io/projected/df410080-3e08-4f8d-b1d3-84db851d8782-kube-api-access-8l87z\") pod \"payload-processing-55fbbbf9dc-885tl\" (UID: \"df410080-3e08-4f8d-b1d3-84db851d8782\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-885tl"
Nov 03 00:13:13.538744 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:13.538723    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-885tl"
Nov 03 00:13:13.830772 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:13.830742    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-885tl"]
Nov 03 00:13:13.832018 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:13:13.831991    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf410080_3e08_4f8d_b1d3_84db851d8782.slice/crio-067955b1bf2fea799b4231531f21e8b6c966590e44e3a1e25a3b644dbe8edb31 WatchSource:0}: Error finding container 067955b1bf2fea799b4231531f21e8b6c966590e44e3a1e25a3b644dbe8edb31: Status 404 returned error can't find the container with id 067955b1bf2fea799b4231531f21e8b6c966590e44e3a1e25a3b644dbe8edb31
Nov 03 00:13:14.561980 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:14.561942    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-885tl" event={"ID":"df410080-3e08-4f8d-b1d3-84db851d8782","Type":"ContainerStarted","Data":"067955b1bf2fea799b4231531f21e8b6c966590e44e3a1e25a3b644dbe8edb31"}
Nov 03 00:13:16.574493 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:16.574449    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-885tl" event={"ID":"df410080-3e08-4f8d-b1d3-84db851d8782","Type":"ContainerStarted","Data":"51ce4bc91216d66f795a38a60af249fef168c1bffc3b3b670aad38674e602941"}
Nov 03 00:13:16.574868 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:16.574530    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-885tl"
Nov 03 00:13:16.607022 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:16.606969    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-885tl" podStartSLOduration=1.652652265 podStartE2EDuration="3.606954184s" totalImagesPullingTime="1.954301919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:16.606682318 +0000 UTC m=+479.486266358" watchObservedRunningTime="2026-11-03 00:13:16.606954184 +0000 UTC m=+479.486538224"
Nov 03 00:13:27.582687 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:27.582658    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-885tl"
Nov 03 00:13:37.362320 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:13:37.362280    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:13:46.784080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:46.784039    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"]
Nov 03 00:13:46.787841 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:46.787816    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"
Nov 03 00:13:46.800714 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:46.800692    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fm44c\""
Nov 03 00:13:46.810099 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:46.810061    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 03 00:13:46.810099 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:46.810068    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 00:13:46.810099 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:46.810098    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 00:13:46.812751 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:46.812724    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"]
Nov 03 00:13:46.841691 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:46.841671    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beb27702-027a-4c8a-9789-2a59dcf2fb56-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q2wkw\" (UID: \"beb27702-027a-4c8a-9789-2a59dcf2fb56\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"
Nov 03 00:13:46.841791 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:46.841713    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxv82\" (UniqueName: \"kubernetes.io/projected/beb27702-027a-4c8a-9789-2a59dcf2fb56-kube-api-access-lxv82\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q2wkw\" (UID: \"beb27702-027a-4c8a-9789-2a59dcf2fb56\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"
Nov 03 00:13:46.841833 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:46.841795    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beb27702-027a-4c8a-9789-2a59dcf2fb56-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q2wkw\" (UID: \"beb27702-027a-4c8a-9789-2a59dcf2fb56\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"
Nov 03 00:13:46.841895 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:46.841878    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb27702-027a-4c8a-9789-2a59dcf2fb56-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q2wkw\" (UID: \"beb27702-027a-4c8a-9789-2a59dcf2fb56\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"
Nov 03 00:13:46.841955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:46.841928    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb27702-027a-4c8a-9789-2a59dcf2fb56-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q2wkw\" (UID: \"beb27702-027a-4c8a-9789-2a59dcf2fb56\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"
Nov 03 00:13:46.841991 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:46.841955    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb27702-027a-4c8a-9789-2a59dcf2fb56-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q2wkw\" (UID: \"beb27702-027a-4c8a-9789-2a59dcf2fb56\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"
Nov 03 00:13:46.943159 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:46.943133    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb27702-027a-4c8a-9789-2a59dcf2fb56-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q2wkw\" (UID: \"beb27702-027a-4c8a-9789-2a59dcf2fb56\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"
Nov 03 00:13:46.943249 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:46.943168    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb27702-027a-4c8a-9789-2a59dcf2fb56-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q2wkw\" (UID: \"beb27702-027a-4c8a-9789-2a59dcf2fb56\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"
Nov 03 00:13:46.943249 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:46.943190    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb27702-027a-4c8a-9789-2a59dcf2fb56-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q2wkw\" (UID: \"beb27702-027a-4c8a-9789-2a59dcf2fb56\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"
Nov 03 00:13:46.943249 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:46.943212    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beb27702-027a-4c8a-9789-2a59dcf2fb56-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q2wkw\" (UID: \"beb27702-027a-4c8a-9789-2a59dcf2fb56\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"
Nov 03 00:13:46.943249 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:46.943238    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxv82\" (UniqueName: \"kubernetes.io/projected/beb27702-027a-4c8a-9789-2a59dcf2fb56-kube-api-access-lxv82\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q2wkw\" (UID: \"beb27702-027a-4c8a-9789-2a59dcf2fb56\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"
Nov 03 00:13:46.943435 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:46.943271    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beb27702-027a-4c8a-9789-2a59dcf2fb56-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q2wkw\" (UID: \"beb27702-027a-4c8a-9789-2a59dcf2fb56\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"
Nov 03 00:13:46.943593 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:46.943570    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb27702-027a-4c8a-9789-2a59dcf2fb56-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q2wkw\" (UID: \"beb27702-027a-4c8a-9789-2a59dcf2fb56\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"
Nov 03 00:13:46.943657 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:46.943618    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beb27702-027a-4c8a-9789-2a59dcf2fb56-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q2wkw\" (UID: \"beb27702-027a-4c8a-9789-2a59dcf2fb56\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"
Nov 03 00:13:46.943657 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:46.943640    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beb27702-027a-4c8a-9789-2a59dcf2fb56-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q2wkw\" (UID: \"beb27702-027a-4c8a-9789-2a59dcf2fb56\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"
Nov 03 00:13:46.945348 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:46.945328    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb27702-027a-4c8a-9789-2a59dcf2fb56-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q2wkw\" (UID: \"beb27702-027a-4c8a-9789-2a59dcf2fb56\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"
Nov 03 00:13:46.945649 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:46.945627    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb27702-027a-4c8a-9789-2a59dcf2fb56-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q2wkw\" (UID: \"beb27702-027a-4c8a-9789-2a59dcf2fb56\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"
Nov 03 00:13:46.960711 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:46.960684    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxv82\" (UniqueName: \"kubernetes.io/projected/beb27702-027a-4c8a-9789-2a59dcf2fb56-kube-api-access-lxv82\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q2wkw\" (UID: \"beb27702-027a-4c8a-9789-2a59dcf2fb56\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"
Nov 03 00:13:47.102577 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:47.102547    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"
Nov 03 00:13:47.395998 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:47.395971    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"]
Nov 03 00:13:47.397804 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:13:47.397780    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb27702_027a_4c8a_9789_2a59dcf2fb56.slice/crio-8f75d72453669a3146a7c7a7371c679e9111e87264cf3ecd14432f04f5fab69c WatchSource:0}: Error finding container 8f75d72453669a3146a7c7a7371c679e9111e87264cf3ecd14432f04f5fab69c: Status 404 returned error can't find the container with id 8f75d72453669a3146a7c7a7371c679e9111e87264cf3ecd14432f04f5fab69c
Nov 03 00:13:47.750341 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:47.750181    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw" event={"ID":"beb27702-027a-4c8a-9789-2a59dcf2fb56","Type":"ContainerStarted","Data":"8f75d72453669a3146a7c7a7371c679e9111e87264cf3ecd14432f04f5fab69c"}
Nov 03 00:13:48.758323 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:48.758291    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw" event={"ID":"beb27702-027a-4c8a-9789-2a59dcf2fb56","Type":"ContainerStarted","Data":"66b0915e28c30b839e998f31339d8113f993ec8a29267dfacebd816946ec4d9b"}
Nov 03 00:13:48.791741 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:48.791699    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw" podStartSLOduration=1.527765086 podStartE2EDuration="2.791685201s" totalImagesPullingTime="1.263920115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:48.790855787 +0000 UTC m=+511.670439830" watchObservedRunningTime="2026-11-03 00:13:48.791685201 +0000 UTC m=+511.671269245"
Nov 03 00:13:49.102764 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:49.102734    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"
Nov 03 00:13:49.324613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:49.324582    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"]
Nov 03 00:13:49.328623 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:49.328597    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"
Nov 03 00:13:49.353220 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:49.353146    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 03 00:13:49.354096 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:49.354070    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"]
Nov 03 00:13:49.373451 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:49.373425    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b908c0c-a41f-474b-ae30-5a3478684a98-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-p94dw\" (UID: \"5b908c0c-a41f-474b-ae30-5a3478684a98\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"
Nov 03 00:13:49.373599 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:49.373456    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spwlr\" (UniqueName: \"kubernetes.io/projected/5b908c0c-a41f-474b-ae30-5a3478684a98-kube-api-access-spwlr\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-p94dw\" (UID: \"5b908c0c-a41f-474b-ae30-5a3478684a98\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"
Nov 03 00:13:49.373599 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:49.373487    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b908c0c-a41f-474b-ae30-5a3478684a98-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-p94dw\" (UID: \"5b908c0c-a41f-474b-ae30-5a3478684a98\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"
Nov 03 00:13:49.373599 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:49.373506    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b908c0c-a41f-474b-ae30-5a3478684a98-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-p94dw\" (UID: \"5b908c0c-a41f-474b-ae30-5a3478684a98\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"
Nov 03 00:13:49.373599 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:49.373521    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b908c0c-a41f-474b-ae30-5a3478684a98-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-p94dw\" (UID: \"5b908c0c-a41f-474b-ae30-5a3478684a98\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"
Nov 03 00:13:49.373599 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:49.373551    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b908c0c-a41f-474b-ae30-5a3478684a98-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-p94dw\" (UID: \"5b908c0c-a41f-474b-ae30-5a3478684a98\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"
Nov 03 00:13:49.474099 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:49.474064    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b908c0c-a41f-474b-ae30-5a3478684a98-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-p94dw\" (UID: \"5b908c0c-a41f-474b-ae30-5a3478684a98\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"
Nov 03 00:13:49.474299 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:49.474172    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b908c0c-a41f-474b-ae30-5a3478684a98-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-p94dw\" (UID: \"5b908c0c-a41f-474b-ae30-5a3478684a98\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"
Nov 03 00:13:49.474299 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:49.474198    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spwlr\" (UniqueName: \"kubernetes.io/projected/5b908c0c-a41f-474b-ae30-5a3478684a98-kube-api-access-spwlr\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-p94dw\" (UID: \"5b908c0c-a41f-474b-ae30-5a3478684a98\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"
Nov 03 00:13:49.474299 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:49.474245    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b908c0c-a41f-474b-ae30-5a3478684a98-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-p94dw\" (UID: \"5b908c0c-a41f-474b-ae30-5a3478684a98\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"
Nov 03 00:13:49.474531 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:49.474372    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b908c0c-a41f-474b-ae30-5a3478684a98-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-p94dw\" (UID: \"5b908c0c-a41f-474b-ae30-5a3478684a98\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"
Nov 03 00:13:49.474531 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:49.474422    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b908c0c-a41f-474b-ae30-5a3478684a98-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-p94dw\" (UID: \"5b908c0c-a41f-474b-ae30-5a3478684a98\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"
Nov 03 00:13:49.474630 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:49.474589    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b908c0c-a41f-474b-ae30-5a3478684a98-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-p94dw\" (UID: \"5b908c0c-a41f-474b-ae30-5a3478684a98\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"
Nov 03 00:13:49.474630 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:49.474596    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b908c0c-a41f-474b-ae30-5a3478684a98-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-p94dw\" (UID: \"5b908c0c-a41f-474b-ae30-5a3478684a98\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"
Nov 03 00:13:49.474801 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:49.474776    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b908c0c-a41f-474b-ae30-5a3478684a98-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-p94dw\" (UID: \"5b908c0c-a41f-474b-ae30-5a3478684a98\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"
Nov 03 00:13:49.476645 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:49.476613    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b908c0c-a41f-474b-ae30-5a3478684a98-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-p94dw\" (UID: \"5b908c0c-a41f-474b-ae30-5a3478684a98\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"
Nov 03 00:13:49.476774 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:49.476697    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b908c0c-a41f-474b-ae30-5a3478684a98-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-p94dw\" (UID: \"5b908c0c-a41f-474b-ae30-5a3478684a98\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"
Nov 03 00:13:49.491395 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:49.491368    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spwlr\" (UniqueName: \"kubernetes.io/projected/5b908c0c-a41f-474b-ae30-5a3478684a98-kube-api-access-spwlr\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-p94dw\" (UID: \"5b908c0c-a41f-474b-ae30-5a3478684a98\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"
Nov 03 00:13:49.662120 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:49.662033    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"
Nov 03 00:13:49.971871 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:49.971835    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"]
Nov 03 00:13:49.979932 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:13:49.976253    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b908c0c_a41f_474b_ae30_5a3478684a98.slice/crio-e70ca96a4a11fca3a9a578c60cb4dff99a2f947c0e77da9a78f3e5d6a0c39bbb WatchSource:0}: Error finding container e70ca96a4a11fca3a9a578c60cb4dff99a2f947c0e77da9a78f3e5d6a0c39bbb: Status 404 returned error can't find the container with id e70ca96a4a11fca3a9a578c60cb4dff99a2f947c0e77da9a78f3e5d6a0c39bbb
Nov 03 00:13:50.772980 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:50.772946    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw" event={"ID":"5b908c0c-a41f-474b-ae30-5a3478684a98","Type":"ContainerStarted","Data":"996028f39f66929bc2257c7724126b8853a90ef0b0e839f33e74135453aed272"}
Nov 03 00:13:50.773169 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:50.773008    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw" event={"ID":"5b908c0c-a41f-474b-ae30-5a3478684a98","Type":"ContainerStarted","Data":"e70ca96a4a11fca3a9a578c60cb4dff99a2f947c0e77da9a78f3e5d6a0c39bbb"}
Nov 03 00:13:50.807801 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:50.806664    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw" podStartSLOduration=1.603325586 podStartE2EDuration="1.806647423s" totalImagesPullingTime="203.321837ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:50.806019331 +0000 UTC m=+513.685603375" watchObservedRunningTime="2026-11-03 00:13:50.806647423 +0000 UTC m=+513.686231469"
Nov 03 00:13:51.283038 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:51.282996    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"]
Nov 03 00:13:51.287600 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:51.287571    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"
Nov 03 00:13:51.311268 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:51.309198    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 03 00:13:51.316563 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:51.316509    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"]
Nov 03 00:13:51.391938 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:51.391894    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61a532fa-57a2-486e-a29b-474f5f77cce2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7\" (UID: \"61a532fa-57a2-486e-a29b-474f5f77cce2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"
Nov 03 00:13:51.392198 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:51.392166    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5vrw\" (UniqueName: \"kubernetes.io/projected/61a532fa-57a2-486e-a29b-474f5f77cce2-kube-api-access-g5vrw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7\" (UID: \"61a532fa-57a2-486e-a29b-474f5f77cce2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"
Nov 03 00:13:51.392313 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:51.392205    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61a532fa-57a2-486e-a29b-474f5f77cce2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7\" (UID: \"61a532fa-57a2-486e-a29b-474f5f77cce2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"
Nov 03 00:13:51.392313 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:51.392230    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61a532fa-57a2-486e-a29b-474f5f77cce2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7\" (UID: \"61a532fa-57a2-486e-a29b-474f5f77cce2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"
Nov 03 00:13:51.392441 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:51.392364    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61a532fa-57a2-486e-a29b-474f5f77cce2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7\" (UID: \"61a532fa-57a2-486e-a29b-474f5f77cce2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"
Nov 03 00:13:51.392441 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:51.392420    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61a532fa-57a2-486e-a29b-474f5f77cce2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7\" (UID: \"61a532fa-57a2-486e-a29b-474f5f77cce2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"
Nov 03 00:13:51.493505 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:51.493460    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61a532fa-57a2-486e-a29b-474f5f77cce2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7\" (UID: \"61a532fa-57a2-486e-a29b-474f5f77cce2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"
Nov 03 00:13:51.493698 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:51.493569    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61a532fa-57a2-486e-a29b-474f5f77cce2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7\" (UID: \"61a532fa-57a2-486e-a29b-474f5f77cce2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"
Nov 03 00:13:51.493698 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:51.493619    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5vrw\" (UniqueName: \"kubernetes.io/projected/61a532fa-57a2-486e-a29b-474f5f77cce2-kube-api-access-g5vrw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7\" (UID: \"61a532fa-57a2-486e-a29b-474f5f77cce2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"
Nov 03 00:13:51.493698 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:51.493650    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61a532fa-57a2-486e-a29b-474f5f77cce2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7\" (UID: \"61a532fa-57a2-486e-a29b-474f5f77cce2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"
Nov 03 00:13:51.493698 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:51.493675    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61a532fa-57a2-486e-a29b-474f5f77cce2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7\" (UID: \"61a532fa-57a2-486e-a29b-474f5f77cce2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"
Nov 03 00:13:51.493947 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:51.493781    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61a532fa-57a2-486e-a29b-474f5f77cce2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7\" (UID: \"61a532fa-57a2-486e-a29b-474f5f77cce2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"
Nov 03 00:13:51.494022 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:51.493980    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61a532fa-57a2-486e-a29b-474f5f77cce2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7\" (UID: \"61a532fa-57a2-486e-a29b-474f5f77cce2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"
Nov 03 00:13:51.494279 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:51.494250    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61a532fa-57a2-486e-a29b-474f5f77cce2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7\" (UID: \"61a532fa-57a2-486e-a29b-474f5f77cce2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"
Nov 03 00:13:51.494279 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:51.494267    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61a532fa-57a2-486e-a29b-474f5f77cce2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7\" (UID: \"61a532fa-57a2-486e-a29b-474f5f77cce2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"
Nov 03 00:13:51.496433 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:51.496411    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61a532fa-57a2-486e-a29b-474f5f77cce2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7\" (UID: \"61a532fa-57a2-486e-a29b-474f5f77cce2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"
Nov 03 00:13:51.496734 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:51.496707    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61a532fa-57a2-486e-a29b-474f5f77cce2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7\" (UID: \"61a532fa-57a2-486e-a29b-474f5f77cce2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"
Nov 03 00:13:51.511575 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:51.511543    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5vrw\" (UniqueName: \"kubernetes.io/projected/61a532fa-57a2-486e-a29b-474f5f77cce2-kube-api-access-g5vrw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7\" (UID: \"61a532fa-57a2-486e-a29b-474f5f77cce2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"
Nov 03 00:13:51.617983 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:51.617943    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"
Nov 03 00:13:51.663061 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:51.663021    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"
Nov 03 00:13:51.914730 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:51.914698    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"]
Nov 03 00:13:51.915394 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:13:51.915368    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a532fa_57a2_486e_a29b_474f5f77cce2.slice/crio-d9fe423b20da1364bdc61919d24d55cce39aeb2b6c343172728072ca265f4273 WatchSource:0}: Error finding container d9fe423b20da1364bdc61919d24d55cce39aeb2b6c343172728072ca265f4273: Status 404 returned error can't find the container with id d9fe423b20da1364bdc61919d24d55cce39aeb2b6c343172728072ca265f4273
Nov 03 00:13:52.786186 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:52.786149    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7" event={"ID":"61a532fa-57a2-486e-a29b-474f5f77cce2","Type":"ContainerStarted","Data":"5ea26ee29f966cb2497050167e3f40da4f13aedc8cf2d034cdcb61a61c1ac9fa"}
Nov 03 00:13:52.786604 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:52.786209    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7" event={"ID":"61a532fa-57a2-486e-a29b-474f5f77cce2","Type":"ContainerStarted","Data":"d9fe423b20da1364bdc61919d24d55cce39aeb2b6c343172728072ca265f4273"}
Nov 03 00:13:52.821314 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:52.821265    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7" podStartSLOduration=1.645622371 podStartE2EDuration="1.821251006s" totalImagesPullingTime="175.628635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:52.819260556 +0000 UTC m=+515.698844599" watchObservedRunningTime="2026-11-03 00:13:52.821251006 +0000 UTC m=+515.700835094"
Nov 03 00:13:53.618819 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:53.618782    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"
Nov 03 00:13:57.103146 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:57.103104    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"
Nov 03 00:13:57.115747 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:57.115711    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"
Nov 03 00:13:57.840291 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:57.840263    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q2wkw"
Nov 03 00:13:57.966806 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:57.966772    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"]
Nov 03 00:13:57.971065 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:57.971044    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:13:57.982343 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:57.982321    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"]
Nov 03 00:13:57.983146 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:57.983128    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 03 00:13:58.052753 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:58.052713    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21760fa1-e581-4657-b63e-d9dd3dde6084-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h\" (UID: \"21760fa1-e581-4657-b63e-d9dd3dde6084\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:13:58.052903 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:58.052760    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21760fa1-e581-4657-b63e-d9dd3dde6084-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h\" (UID: \"21760fa1-e581-4657-b63e-d9dd3dde6084\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:13:58.052903 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:58.052829    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21760fa1-e581-4657-b63e-d9dd3dde6084-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h\" (UID: \"21760fa1-e581-4657-b63e-d9dd3dde6084\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:13:58.053011 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:58.052897    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc4tn\" (UniqueName: \"kubernetes.io/projected/21760fa1-e581-4657-b63e-d9dd3dde6084-kube-api-access-bc4tn\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h\" (UID: \"21760fa1-e581-4657-b63e-d9dd3dde6084\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:13:58.053011 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:58.052957    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21760fa1-e581-4657-b63e-d9dd3dde6084-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h\" (UID: \"21760fa1-e581-4657-b63e-d9dd3dde6084\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:13:58.053077 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:58.053036    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21760fa1-e581-4657-b63e-d9dd3dde6084-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h\" (UID: \"21760fa1-e581-4657-b63e-d9dd3dde6084\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:13:58.154314 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:58.154278    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21760fa1-e581-4657-b63e-d9dd3dde6084-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h\" (UID: \"21760fa1-e581-4657-b63e-d9dd3dde6084\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:13:58.154780 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:58.154325    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21760fa1-e581-4657-b63e-d9dd3dde6084-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h\" (UID: \"21760fa1-e581-4657-b63e-d9dd3dde6084\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:13:58.154780 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:58.154343    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc4tn\" (UniqueName: \"kubernetes.io/projected/21760fa1-e581-4657-b63e-d9dd3dde6084-kube-api-access-bc4tn\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h\" (UID: \"21760fa1-e581-4657-b63e-d9dd3dde6084\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:13:58.154780 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:58.154362    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21760fa1-e581-4657-b63e-d9dd3dde6084-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h\" (UID: \"21760fa1-e581-4657-b63e-d9dd3dde6084\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:13:58.154780 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:58.154419    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21760fa1-e581-4657-b63e-d9dd3dde6084-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h\" (UID: \"21760fa1-e581-4657-b63e-d9dd3dde6084\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:13:58.154780 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:58.154465    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21760fa1-e581-4657-b63e-d9dd3dde6084-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h\" (UID: \"21760fa1-e581-4657-b63e-d9dd3dde6084\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:13:58.154780 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:58.154768    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21760fa1-e581-4657-b63e-d9dd3dde6084-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h\" (UID: \"21760fa1-e581-4657-b63e-d9dd3dde6084\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:13:58.155031 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:58.154807    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21760fa1-e581-4657-b63e-d9dd3dde6084-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h\" (UID: \"21760fa1-e581-4657-b63e-d9dd3dde6084\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:13:58.155031 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:58.154826    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21760fa1-e581-4657-b63e-d9dd3dde6084-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h\" (UID: \"21760fa1-e581-4657-b63e-d9dd3dde6084\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:13:58.156631 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:58.156614    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21760fa1-e581-4657-b63e-d9dd3dde6084-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h\" (UID: \"21760fa1-e581-4657-b63e-d9dd3dde6084\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:13:58.156983 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:58.156965    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21760fa1-e581-4657-b63e-d9dd3dde6084-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h\" (UID: \"21760fa1-e581-4657-b63e-d9dd3dde6084\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:13:58.174063 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:58.174042    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc4tn\" (UniqueName: \"kubernetes.io/projected/21760fa1-e581-4657-b63e-d9dd3dde6084-kube-api-access-bc4tn\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h\" (UID: \"21760fa1-e581-4657-b63e-d9dd3dde6084\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:13:58.284795 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:58.284771    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:13:58.599289 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:13:58.599253    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21760fa1_e581_4657_b63e_d9dd3dde6084.slice/crio-47a2688809a3037094da0f079e8bbacdf9bdeb7f65be5f2585fa3e0e7e4b0dba WatchSource:0}: Error finding container 47a2688809a3037094da0f079e8bbacdf9bdeb7f65be5f2585fa3e0e7e4b0dba: Status 404 returned error can't find the container with id 47a2688809a3037094da0f079e8bbacdf9bdeb7f65be5f2585fa3e0e7e4b0dba
Nov 03 00:13:58.599430 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:58.599324    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"]
Nov 03 00:13:58.839167 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:58.839139    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" event={"ID":"21760fa1-e581-4657-b63e-d9dd3dde6084","Type":"ContainerStarted","Data":"47a2688809a3037094da0f079e8bbacdf9bdeb7f65be5f2585fa3e0e7e4b0dba"}
Nov 03 00:13:59.663193 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:59.663162    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"
Nov 03 00:13:59.675602 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:59.675576    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"
Nov 03 00:13:59.846601 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:59.846566    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" event={"ID":"21760fa1-e581-4657-b63e-d9dd3dde6084","Type":"ContainerStarted","Data":"81807b2cc9423baf6905ef96233d95de59def8b0ef7c5fb8c8723ce3155b9bd9"}
Nov 03 00:13:59.858074 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:59.858051    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-p94dw"
Nov 03 00:13:59.885594 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:13:59.885537    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podStartSLOduration=2.703383887 podStartE2EDuration="2.885520252s" totalImagesPullingTime="182.136365ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:59.880274784 +0000 UTC m=+522.759858831" watchObservedRunningTime="2026-11-03 00:13:59.885520252 +0000 UTC m=+522.765104297"
Nov 03 00:14:00.285756 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:00.285719    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:14:01.618649 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:01.618614    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"
Nov 03 00:14:01.637770 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:01.637738    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"
Nov 03 00:14:01.883822 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:01.883746    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7"
Nov 03 00:14:07.362223 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:14:07.362187    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:14:08.285733 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:08.285702    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:14:08.297597 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:08.297572    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:14:08.922265 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:08.922201    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:14:19.040706 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:19.040668    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f5fd7bcdf-cdr4b"]
Nov 03 00:14:19.044817 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:19.044794    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f5fd7bcdf-cdr4b"
Nov 03 00:14:19.059705 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:19.059684    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f5fd7bcdf-cdr4b"]
Nov 03 00:14:19.142222 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:19.142192    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7pqs\" (UniqueName: \"kubernetes.io/projected/3be4bd66-b870-41a8-931d-94f258522d39-kube-api-access-q7pqs\") pod \"authorino-6f5fd7bcdf-cdr4b\" (UID: \"3be4bd66-b870-41a8-931d-94f258522d39\") " pod="kuadrant-system/authorino-6f5fd7bcdf-cdr4b"
Nov 03 00:14:19.142336 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:19.142231    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3be4bd66-b870-41a8-931d-94f258522d39-tls-cert\") pod \"authorino-6f5fd7bcdf-cdr4b\" (UID: \"3be4bd66-b870-41a8-931d-94f258522d39\") " pod="kuadrant-system/authorino-6f5fd7bcdf-cdr4b"
Nov 03 00:14:19.243176 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:19.243153    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7pqs\" (UniqueName: \"kubernetes.io/projected/3be4bd66-b870-41a8-931d-94f258522d39-kube-api-access-q7pqs\") pod \"authorino-6f5fd7bcdf-cdr4b\" (UID: \"3be4bd66-b870-41a8-931d-94f258522d39\") " pod="kuadrant-system/authorino-6f5fd7bcdf-cdr4b"
Nov 03 00:14:19.243290 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:19.243183    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3be4bd66-b870-41a8-931d-94f258522d39-tls-cert\") pod \"authorino-6f5fd7bcdf-cdr4b\" (UID: \"3be4bd66-b870-41a8-931d-94f258522d39\") " pod="kuadrant-system/authorino-6f5fd7bcdf-cdr4b"
Nov 03 00:14:19.245521 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:19.245498    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3be4bd66-b870-41a8-931d-94f258522d39-tls-cert\") pod \"authorino-6f5fd7bcdf-cdr4b\" (UID: \"3be4bd66-b870-41a8-931d-94f258522d39\") " pod="kuadrant-system/authorino-6f5fd7bcdf-cdr4b"
Nov 03 00:14:19.259037 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:19.259010    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7pqs\" (UniqueName: \"kubernetes.io/projected/3be4bd66-b870-41a8-931d-94f258522d39-kube-api-access-q7pqs\") pod \"authorino-6f5fd7bcdf-cdr4b\" (UID: \"3be4bd66-b870-41a8-931d-94f258522d39\") " pod="kuadrant-system/authorino-6f5fd7bcdf-cdr4b"
Nov 03 00:14:19.355824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:19.355799    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f5fd7bcdf-cdr4b"
Nov 03 00:14:19.648789 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:19.648761    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f5fd7bcdf-cdr4b"]
Nov 03 00:14:19.649881 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:14:19.649853    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3be4bd66_b870_41a8_931d_94f258522d39.slice/crio-fb5b78f7881d616ca07031685fba0d4017b499a0b2930c0358c429356c73afe4 WatchSource:0}: Error finding container fb5b78f7881d616ca07031685fba0d4017b499a0b2930c0358c429356c73afe4: Status 404 returned error can't find the container with id fb5b78f7881d616ca07031685fba0d4017b499a0b2930c0358c429356c73afe4
Nov 03 00:14:19.971630 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:19.971545    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f5fd7bcdf-cdr4b" event={"ID":"3be4bd66-b870-41a8-931d-94f258522d39","Type":"ContainerStarted","Data":"fb5b78f7881d616ca07031685fba0d4017b499a0b2930c0358c429356c73afe4"}
Nov 03 00:14:20.981145 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:20.981096    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f5fd7bcdf-cdr4b" event={"ID":"3be4bd66-b870-41a8-931d-94f258522d39","Type":"ContainerStarted","Data":"680c9f95beffe87d07c5953d47285ff986e37acd5db7ed15854126f366f1c1de"}
Nov 03 00:14:21.021931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:21.020056    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f5fd7bcdf-cdr4b" podStartSLOduration=1.444078557 podStartE2EDuration="2.020039409s" totalImagesPullingTime="575.960852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:14:21.017350468 +0000 UTC m=+543.896934511" watchObservedRunningTime="2026-11-03 00:14:21.020039409 +0000 UTC m=+543.899623453"
Nov 03 00:14:21.052222 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:21.049392    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56bf98447d-nrnvg"]
Nov 03 00:14:21.052222 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:21.049620    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56bf98447d-nrnvg" podUID="7a662a32-1c38-4a1a-a49a-b2b90940582e" containerName="authorino" containerID="cri-o://6606bfe28e51df645a82b400ac338de44320302dc4be6099c957cc18a52ecf31" gracePeriod=30
Nov 03 00:14:21.324389 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:21.324356    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56bf98447d-nrnvg"
Nov 03 00:14:21.463487 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:21.463400    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a662a32-1c38-4a1a-a49a-b2b90940582e-kube-api-access-fkd9k\" (UniqueName: \"kubernetes.io/projected/7a662a32-1c38-4a1a-a49a-b2b90940582e-kube-api-access-fkd9k\") pod \"7a662a32-1c38-4a1a-a49a-b2b90940582e\" (UID: \"7a662a32-1c38-4a1a-a49a-b2b90940582e\") "
Nov 03 00:14:21.463655 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:21.463604    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a662a32-1c38-4a1a-a49a-b2b90940582e-tls-cert\" (UniqueName: \"kubernetes.io/secret/7a662a32-1c38-4a1a-a49a-b2b90940582e-tls-cert\") pod \"7a662a32-1c38-4a1a-a49a-b2b90940582e\" (UID: \"7a662a32-1c38-4a1a-a49a-b2b90940582e\") "
Nov 03 00:14:21.465667 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:21.465622    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a662a32-1c38-4a1a-a49a-b2b90940582e-kube-api-access-fkd9k" pod "7a662a32-1c38-4a1a-a49a-b2b90940582e" (UID: "7a662a32-1c38-4a1a-a49a-b2b90940582e"). InnerVolumeSpecName "kube-api-access-fkd9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:14:21.475381 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:21.475348    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a662a32-1c38-4a1a-a49a-b2b90940582e-tls-cert" pod "7a662a32-1c38-4a1a-a49a-b2b90940582e" (UID: "7a662a32-1c38-4a1a-a49a-b2b90940582e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:14:21.564364 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:21.564287    5012 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7a662a32-1c38-4a1a-a49a-b2b90940582e-tls-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:14:21.564364 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:21.564319    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkd9k\" (UniqueName: \"kubernetes.io/projected/7a662a32-1c38-4a1a-a49a-b2b90940582e-kube-api-access-fkd9k\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:14:21.989003 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:21.988963    5012 generic.go:356] "Generic (PLEG): container finished" podID="7a662a32-1c38-4a1a-a49a-b2b90940582e" containerID="6606bfe28e51df645a82b400ac338de44320302dc4be6099c957cc18a52ecf31" exitCode=0
Nov 03 00:14:21.989501 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:21.989013    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56bf98447d-nrnvg"
Nov 03 00:14:21.989501 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:21.989052    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56bf98447d-nrnvg" event={"ID":"7a662a32-1c38-4a1a-a49a-b2b90940582e","Type":"ContainerDied","Data":"6606bfe28e51df645a82b400ac338de44320302dc4be6099c957cc18a52ecf31"}
Nov 03 00:14:21.989501 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:21.989099    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56bf98447d-nrnvg" event={"ID":"7a662a32-1c38-4a1a-a49a-b2b90940582e","Type":"ContainerDied","Data":"c7dda050089e2d0745fe59abf3a960f853e82bc4c3a2c52ed383787ffcd91065"}
Nov 03 00:14:21.989501 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:21.989120    5012 scope.go:122] "RemoveContainer" containerID="6606bfe28e51df645a82b400ac338de44320302dc4be6099c957cc18a52ecf31"
Nov 03 00:14:22.000764 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:22.000738    5012 scope.go:122] "RemoveContainer" containerID="6606bfe28e51df645a82b400ac338de44320302dc4be6099c957cc18a52ecf31"
Nov 03 00:14:22.001132 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:14:22.001106    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6606bfe28e51df645a82b400ac338de44320302dc4be6099c957cc18a52ecf31\": container with ID starting with 6606bfe28e51df645a82b400ac338de44320302dc4be6099c957cc18a52ecf31 not found: ID does not exist" containerID="6606bfe28e51df645a82b400ac338de44320302dc4be6099c957cc18a52ecf31"
Nov 03 00:14:22.001227 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:22.001143    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6606bfe28e51df645a82b400ac338de44320302dc4be6099c957cc18a52ecf31"} err="failed to get container status \"6606bfe28e51df645a82b400ac338de44320302dc4be6099c957cc18a52ecf31\": rpc error: code = NotFound desc = could not find container \"6606bfe28e51df645a82b400ac338de44320302dc4be6099c957cc18a52ecf31\": container with ID starting with 6606bfe28e51df645a82b400ac338de44320302dc4be6099c957cc18a52ecf31 not found: ID does not exist"
Nov 03 00:14:22.039666 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:22.039635    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56bf98447d-nrnvg"]
Nov 03 00:14:22.048086 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:22.048058    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56bf98447d-nrnvg"]
Nov 03 00:14:23.244291 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:23.244249    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a662a32-1c38-4a1a-a49a-b2b90940582e" path="/var/lib/kubelet/pods/7a662a32-1c38-4a1a-a49a-b2b90940582e/volumes"
Nov 03 00:14:27.249069 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.249017    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"]
Nov 03 00:14:27.249801 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.249778    5012 cpu_manager.go:414] "containerMap: removing container" podUID="7a662a32-1c38-4a1a-a49a-b2b90940582e" containerName="authorino"
Nov 03 00:14:27.249894 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.249801    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a662a32-1c38-4a1a-a49a-b2b90940582e" containerName="authorino"
Nov 03 00:14:27.250060 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.250046    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a662a32-1c38-4a1a-a49a-b2b90940582e" containerName="authorino"
Nov 03 00:14:27.255324 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.255300    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.268644 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.268620    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"]
Nov 03 00:14:27.442927 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.442878    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ggjn\" (UniqueName: \"kubernetes.io/projected/aa430510-e80c-4743-b4f3-838ce423e442-kube-api-access-4ggjn\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.443064 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.442951    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa430510-e80c-4743-b4f3-838ce423e442-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.443064 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.443043    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.443135 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.443066    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.443135 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.443083    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa430510-e80c-4743-b4f3-838ce423e442-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.443203 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.443167    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.443235 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.443211    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.443286 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.443264    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.443324 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.443296    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa430510-e80c-4743-b4f3-838ce423e442-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.544130 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.544066    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa430510-e80c-4743-b4f3-838ce423e442-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.544130 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.544123    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.544297 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.544141    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.544297 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.544157    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa430510-e80c-4743-b4f3-838ce423e442-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.544297 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.544188    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.544297 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.544216    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.544497 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.544355    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.544497 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.544409    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa430510-e80c-4743-b4f3-838ce423e442-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.544497 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.544446    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ggjn\" (UniqueName: \"kubernetes.io/projected/aa430510-e80c-4743-b4f3-838ce423e442-kube-api-access-4ggjn\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.544686 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.544660    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.544686 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.544677    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.544873 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.544781    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.544873 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.544826    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.545016 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.544999    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa430510-e80c-4743-b4f3-838ce423e442-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.546475 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.546457    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.546833 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.546816    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa430510-e80c-4743-b4f3-838ce423e442-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.560843 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.560819    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa430510-e80c-4743-b4f3-838ce423e442-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.560947 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.560889    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ggjn\" (UniqueName: \"kubernetes.io/projected/aa430510-e80c-4743-b4f3-838ce423e442-kube-api-access-4ggjn\") pod \"maas-default-gateway-openshift-default-6cc458d887-5lnf2\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.568820 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.568802    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:27.877670 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.877645    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"]
Nov 03 00:14:27.878744 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:14:27.878722    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa430510_e80c_4743_b4f3_838ce423e442.slice/crio-0f5a922ec1e8c60a53cd5cda0fc60e5cfd1a26dc861cf2596d498bb5502c6776 WatchSource:0}: Error finding container 0f5a922ec1e8c60a53cd5cda0fc60e5cfd1a26dc861cf2596d498bb5502c6776: Status 404 returned error can't find the container with id 0f5a922ec1e8c60a53cd5cda0fc60e5cfd1a26dc861cf2596d498bb5502c6776
Nov 03 00:14:27.880987 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.880953    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:14:27.881112 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.881028    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:14:27.881112 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:27.881072    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:14:28.030847 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:28.030810    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2" event={"ID":"aa430510-e80c-4743-b4f3-838ce423e442","Type":"ContainerStarted","Data":"521e4c4e5517f7b94f7342f12d04e7dd45fa3017b977625cf1cb8ea97a4e08e0"}
Nov 03 00:14:28.031029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:28.030872    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2" event={"ID":"aa430510-e80c-4743-b4f3-838ce423e442","Type":"ContainerStarted","Data":"0f5a922ec1e8c60a53cd5cda0fc60e5cfd1a26dc861cf2596d498bb5502c6776"}
Nov 03 00:14:28.069060 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:28.068999    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2" podStartSLOduration=1.06898173 podStartE2EDuration="1.06898173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:14:28.067139194 +0000 UTC m=+550.946723236" watchObservedRunningTime="2026-11-03 00:14:28.06898173 +0000 UTC m=+550.948565774"
Nov 03 00:14:28.570006 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:28.569974    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:28.571394 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:28.571372    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-5lnf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:14:28.571455 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:28.571419    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2" podUID="aa430510-e80c-4743-b4f3-838ce423e442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 03 00:14:29.569746 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:29.569716    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-5lnf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:14:29.569892 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:29.569764    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2" podUID="aa430510-e80c-4743-b4f3-838ce423e442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 03 00:14:30.570028 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:30.569988    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-5lnf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:14:30.570435 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:30.570082    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2" podUID="aa430510-e80c-4743-b4f3-838ce423e442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 03 00:14:31.569161 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:31.569119    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-5lnf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:14:31.569327 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:31.569183    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2" podUID="aa430510-e80c-4743-b4f3-838ce423e442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 03 00:14:32.569468 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:32.569435    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-5lnf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:14:32.569845 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:32.569486    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2" podUID="aa430510-e80c-4743-b4f3-838ce423e442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 03 00:14:33.570129 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:33.570094    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-5lnf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:14:33.570542 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:33.570148    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2" podUID="aa430510-e80c-4743-b4f3-838ce423e442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 03 00:14:34.570171 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:34.570135    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-5lnf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:14:34.570643 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:34.570186    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2" podUID="aa430510-e80c-4743-b4f3-838ce423e442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 03 00:14:36.138456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:36.138421    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:36.138966 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:36.138807    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:36.139646 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:36.139621    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:14:37.361840 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:14:37.361807    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:14:59.270961 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.270869    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"]
Nov 03 00:14:59.275891 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.275872    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.305141 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.305111    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c600f063-openshift-default-dockercfg-xs98b\""
Nov 03 00:14:59.307968 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.307945    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"]
Nov 03 00:14:59.329175 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.329147    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-workload-certs\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.329274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.329176    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-envoy\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.329274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.329204    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-token\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.329345 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.329327    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-data\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.329379 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.329348    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-podinfo\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.329379 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.329364    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-credential-socket\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.329504 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.329488    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-workload-socket\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.329539 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.329512    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjgxv\" (UniqueName: \"kubernetes.io/projected/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-kube-api-access-sjgxv\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.329539 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.329535    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istiod-ca-cert\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.430995 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.430965    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-workload-certs\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.430995 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.430995    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-envoy\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.431227 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.431021    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-token\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.431227 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.431120    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-data\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.431227 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.431156    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-podinfo\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.431227 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.431176    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-credential-socket\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.431453 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.431242    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-workload-socket\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.431453 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.431267    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjgxv\" (UniqueName: \"kubernetes.io/projected/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-kube-api-access-sjgxv\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.431453 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.431307    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istiod-ca-cert\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.431610 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.431490    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-workload-certs\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.431610 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.431517    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-data\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.431610 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.431575    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-credential-socket\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.431779 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.431636    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-workload-socket\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.432058 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.432036    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istiod-ca-cert\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.433144 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.433125    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-envoy\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.434024 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.433999    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-podinfo\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.448013 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.447993    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-token\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.451091 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.451065    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjgxv\" (UniqueName: \"kubernetes.io/projected/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-kube-api-access-sjgxv\") pod \"e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") " pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.590014 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.589949    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:14:59.737333 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.737302    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"]
Nov 03 00:14:59.741295 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.741275    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.758618 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.758599    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-b4cc5943-gw-openshift-default-dockercfg-8lszk\""
Nov 03 00:14:59.762334 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.762310    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"]
Nov 03 00:14:59.835380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.835348    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-data\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.835495 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.835385    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-envoy\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.835538 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.835518    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-token\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.835584 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.835566    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-podinfo\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.835685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.835669    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97953a4d-b832-4282-b010-d06cc1e7e3c2-istiod-ca-cert\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.835750 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.835735    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk6z9\" (UniqueName: \"kubernetes.io/projected/97953a4d-b832-4282-b010-d06cc1e7e3c2-kube-api-access-hk6z9\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.835808 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.835776    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-workload-socket\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.835846 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.835823    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-credential-socket\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.835879 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.835857    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-workload-certs\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.900080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.900056    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"]
Nov 03 00:14:59.901452 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:14:59.901423    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d6a6d6_7eb1_40f1_83d3_547a3299fbc8.slice/crio-baa6c7de5ad0368dfb7f2c23aa03e5d8ea7dc5f7a2055e66ce877f939516c9d7 WatchSource:0}: Error finding container baa6c7de5ad0368dfb7f2c23aa03e5d8ea7dc5f7a2055e66ce877f939516c9d7: Status 404 returned error can't find the container with id baa6c7de5ad0368dfb7f2c23aa03e5d8ea7dc5f7a2055e66ce877f939516c9d7
Nov 03 00:14:59.903734 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.903703    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:14:59.903809 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.903762    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:14:59.903809 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.903791    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:14:59.937245 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.937218    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-data\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.937397 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.937249    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-envoy\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.937397 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.937273    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-token\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.937397 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.937301    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-podinfo\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.937397 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.937377    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97953a4d-b832-4282-b010-d06cc1e7e3c2-istiod-ca-cert\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.937633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.937433    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk6z9\" (UniqueName: \"kubernetes.io/projected/97953a4d-b832-4282-b010-d06cc1e7e3c2-kube-api-access-hk6z9\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.937633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.937468    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-workload-socket\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.937633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.937491    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-credential-socket\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.937633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.937520    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-workload-certs\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.937844 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.937633    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-data\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.937938 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.937833    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-workload-socket\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.937938 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.937851    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-credential-socket\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.938043 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.937971    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-workload-certs\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.938135 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.938118    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97953a4d-b832-4282-b010-d06cc1e7e3c2-istiod-ca-cert\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.939757 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.939737    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-envoy\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.939922 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.939887    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-podinfo\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.953819 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.953793    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-token\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:14:59.954087 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:14:59.954071    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk6z9\" (UniqueName: \"kubernetes.io/projected/97953a4d-b832-4282-b010-d06cc1e7e3c2-kube-api-access-hk6z9\") pod \"e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") " pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:15:00.056006 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:00.055974    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:15:00.142146 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:00.142105    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd"]
Nov 03 00:15:00.146841 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:00.146820    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd"
Nov 03 00:15:00.159894 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:00.159866    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 00:15:00.160049 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:00.160020    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 00:15:00.160145 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:00.160129    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:15:00.163558 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:00.163185    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd"]
Nov 03 00:15:00.235084 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:00.235034    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z" event={"ID":"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8","Type":"ContainerStarted","Data":"1cb8593e890e50a46b1b1df8ece05acf229d7e7740908ba472b6785a215be084"}
Nov 03 00:15:00.235269 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:00.235104    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z" event={"ID":"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8","Type":"ContainerStarted","Data":"baa6c7de5ad0368dfb7f2c23aa03e5d8ea7dc5f7a2055e66ce877f939516c9d7"}
Nov 03 00:15:00.239904 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:00.239880    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894415-l2kjd\" (UID: \"aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd"
Nov 03 00:15:00.240076 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:00.240060    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f74wc\" (UniqueName: \"kubernetes.io/projected/aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e-kube-api-access-f74wc\") pod \"maas-api-key-cleanup-29894415-l2kjd\" (UID: \"aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd"
Nov 03 00:15:00.275046 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:00.275006    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z" podStartSLOduration=1.274994878 podStartE2EDuration="1.274994878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:00.273134881 +0000 UTC m=+583.152718923" watchObservedRunningTime="2026-11-03 00:15:00.274994878 +0000 UTC m=+583.154578920"
Nov 03 00:15:00.341268 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:00.341244    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f74wc\" (UniqueName: \"kubernetes.io/projected/aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e-kube-api-access-f74wc\") pod \"maas-api-key-cleanup-29894415-l2kjd\" (UID: \"aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd"
Nov 03 00:15:00.341646 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:00.341621    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894415-l2kjd\" (UID: \"aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd"
Nov 03 00:15:00.342196 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:00.342176    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894415-l2kjd\" (UID: \"aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd"
Nov 03 00:15:00.357276 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:00.357252    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f74wc\" (UniqueName: \"kubernetes.io/projected/aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e-kube-api-access-f74wc\") pod \"maas-api-key-cleanup-29894415-l2kjd\" (UID: \"aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd"
Nov 03 00:15:00.372443 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:00.372422    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"]
Nov 03 00:15:00.373896 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:15:00.373875    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97953a4d_b832_4282_b010_d06cc1e7e3c2.slice/crio-25c8bf3d0ad1ba0f0740ce70b597aefe20332e107376f1ae4a8c8de37b4c28dc WatchSource:0}: Error finding container 25c8bf3d0ad1ba0f0740ce70b597aefe20332e107376f1ae4a8c8de37b4c28dc: Status 404 returned error can't find the container with id 25c8bf3d0ad1ba0f0740ce70b597aefe20332e107376f1ae4a8c8de37b4c28dc
Nov 03 00:15:00.375966 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:00.375942    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:15:00.376032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:00.375995    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:15:00.376032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:00.376025    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:15:00.463611 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:00.463582    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd"
Nov 03 00:15:00.590734 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:00.590700    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:15:00.595097 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:00.595074    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:15:00.776379 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:00.776344    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd"]
Nov 03 00:15:00.777837 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:15:00.777803    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae16b30_ee5b_45f7_8d0e_ac5dda79bb8e.slice/crio-67bcadff723ddcb7ea7705e1d08d5e1c8ad0b71809a4d34c52b8f30e125a6d98 WatchSource:0}: Error finding container 67bcadff723ddcb7ea7705e1d08d5e1c8ad0b71809a4d34c52b8f30e125a6d98: Status 404 returned error can't find the container with id 67bcadff723ddcb7ea7705e1d08d5e1c8ad0b71809a4d34c52b8f30e125a6d98
Nov 03 00:15:01.248730 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:01.248693    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd" event={"ID":"aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e","Type":"ContainerStarted","Data":"67bcadff723ddcb7ea7705e1d08d5e1c8ad0b71809a4d34c52b8f30e125a6d98"}
Nov 03 00:15:01.250139 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:01.250108    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv" event={"ID":"97953a4d-b832-4282-b010-d06cc1e7e3c2","Type":"ContainerStarted","Data":"6af6579a784d160b63a9ac72aabf7df7d1992c6e4a6a3505ce8607ed34d68cea"}
Nov 03 00:15:01.250271 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:01.250156    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv" event={"ID":"97953a4d-b832-4282-b010-d06cc1e7e3c2","Type":"ContainerStarted","Data":"25c8bf3d0ad1ba0f0740ce70b597aefe20332e107376f1ae4a8c8de37b4c28dc"}
Nov 03 00:15:01.250437 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:01.250418    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:15:01.251363 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:01.251346    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:15:01.287844 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:01.287801    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv" podStartSLOduration=2.287789038 podStartE2EDuration="2.287789038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:01.284961458 +0000 UTC m=+584.164545513" watchObservedRunningTime="2026-11-03 00:15:01.287789038 +0000 UTC m=+584.167373081"
Nov 03 00:15:02.056972 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:02.056935    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:15:02.062693 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:02.062665    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:15:02.255937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:02.255875    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:15:02.256581 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:02.256559    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:15:03.855987 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:03.855959    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz"]
Nov 03 00:15:03.861121 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:03.861104    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz"
Nov 03 00:15:03.880027 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:03.880006    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz"]
Nov 03 00:15:03.885887 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:03.885866    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c600f063\""
Nov 03 00:15:03.886026 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:03.886010    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c600f063-dockercfg-868df\""
Nov 03 00:15:03.951529 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:03.951490    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv"]
Nov 03 00:15:03.955435 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:03.955405    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv"
Nov 03 00:15:03.971355 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:03.971327    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv"]
Nov 03 00:15:03.976566 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:03.976541    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rltgd\" (UniqueName: \"kubernetes.io/projected/4d57748d-abae-487e-8e34-47ff716921be-kube-api-access-rltgd\") pod \"payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz\" (UID: \"4d57748d-abae-487e-8e34-47ff716921be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz"
Nov 03 00:15:03.976664 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:03.976597    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d57748d-abae-487e-8e34-47ff716921be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz\" (UID: \"4d57748d-abae-487e-8e34-47ff716921be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz"
Nov 03 00:15:04.077763 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:04.077694    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d57748d-abae-487e-8e34-47ff716921be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz\" (UID: \"4d57748d-abae-487e-8e34-47ff716921be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz"
Nov 03 00:15:04.077891 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:04.077789    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff82c8b4-dd1a-4870-829e-a37c98bf0280-plugins-config-volume\") pod \"payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv\" (UID: \"ff82c8b4-dd1a-4870-829e-a37c98bf0280\") " pod="openshift-ingress/payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv"
Nov 03 00:15:04.077891 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:04.077867    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rltgd\" (UniqueName: \"kubernetes.io/projected/4d57748d-abae-487e-8e34-47ff716921be-kube-api-access-rltgd\") pod \"payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz\" (UID: \"4d57748d-abae-487e-8e34-47ff716921be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz"
Nov 03 00:15:04.078023 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:04.077903    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr6hf\" (UniqueName: \"kubernetes.io/projected/ff82c8b4-dd1a-4870-829e-a37c98bf0280-kube-api-access-xr6hf\") pod \"payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv\" (UID: \"ff82c8b4-dd1a-4870-829e-a37c98bf0280\") " pod="openshift-ingress/payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv"
Nov 03 00:15:04.078415 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:04.078395    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d57748d-abae-487e-8e34-47ff716921be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz\" (UID: \"4d57748d-abae-487e-8e34-47ff716921be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz"
Nov 03 00:15:04.094839 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:04.094818    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rltgd\" (UniqueName: \"kubernetes.io/projected/4d57748d-abae-487e-8e34-47ff716921be-kube-api-access-rltgd\") pod \"payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz\" (UID: \"4d57748d-abae-487e-8e34-47ff716921be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz"
Nov 03 00:15:04.173508 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:04.173488    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz"
Nov 03 00:15:04.178807 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:04.178784    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff82c8b4-dd1a-4870-829e-a37c98bf0280-plugins-config-volume\") pod \"payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv\" (UID: \"ff82c8b4-dd1a-4870-829e-a37c98bf0280\") " pod="openshift-ingress/payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv"
Nov 03 00:15:04.178902 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:04.178846    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr6hf\" (UniqueName: \"kubernetes.io/projected/ff82c8b4-dd1a-4870-829e-a37c98bf0280-kube-api-access-xr6hf\") pod \"payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv\" (UID: \"ff82c8b4-dd1a-4870-829e-a37c98bf0280\") " pod="openshift-ingress/payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv"
Nov 03 00:15:04.179574 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:04.179550    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff82c8b4-dd1a-4870-829e-a37c98bf0280-plugins-config-volume\") pod \"payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv\" (UID: \"ff82c8b4-dd1a-4870-829e-a37c98bf0280\") " pod="openshift-ingress/payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv"
Nov 03 00:15:04.196805 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:04.196774    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr6hf\" (UniqueName: \"kubernetes.io/projected/ff82c8b4-dd1a-4870-829e-a37c98bf0280-kube-api-access-xr6hf\") pod \"payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv\" (UID: \"ff82c8b4-dd1a-4870-829e-a37c98bf0280\") " pod="openshift-ingress/payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv"
Nov 03 00:15:04.269878 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:04.269818    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd" event={"ID":"aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e","Type":"ContainerStarted","Data":"7485d53c5a1ad975f6100faf0295a26d95c97263ae65b3c2cb290dc2ee15c8cb"}
Nov 03 00:15:04.270419 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:04.270398    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv"
Nov 03 00:15:04.315635 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:04.315571    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd" podStartSLOduration=1.312037377 podStartE2EDuration="4.315550999s" totalImagesPullingTime="3.003513622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:04.311443972 +0000 UTC m=+587.191028016" watchObservedRunningTime="2026-11-03 00:15:04.315550999 +0000 UTC m=+587.195135043"
Nov 03 00:15:04.476615 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:04.476591    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz"]
Nov 03 00:15:04.477956 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:15:04.477933    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d57748d_abae_487e_8e34_47ff716921be.slice/crio-032a615be4e45cdef4bc5c15244e6890e750751cb080fe61f42a098ab372d7a6 WatchSource:0}: Error finding container 032a615be4e45cdef4bc5c15244e6890e750751cb080fe61f42a098ab372d7a6: Status 404 returned error can't find the container with id 032a615be4e45cdef4bc5c15244e6890e750751cb080fe61f42a098ab372d7a6
Nov 03 00:15:04.574012 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:04.573987    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv"]
Nov 03 00:15:04.575411 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:15:04.575388    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff82c8b4_dd1a_4870_829e_a37c98bf0280.slice/crio-dc7defd5d0b3d7c080f37c2e3fde352db143da9a70550326c95c4d9cf05a1116 WatchSource:0}: Error finding container dc7defd5d0b3d7c080f37c2e3fde352db143da9a70550326c95c4d9cf05a1116: Status 404 returned error can't find the container with id dc7defd5d0b3d7c080f37c2e3fde352db143da9a70550326c95c4d9cf05a1116
Nov 03 00:15:05.277617 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.277580    5012 generic.go:356] "Generic (PLEG): container finished" podID="aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e" containerID="7485d53c5a1ad975f6100faf0295a26d95c97263ae65b3c2cb290dc2ee15c8cb" exitCode=60
Nov 03 00:15:05.278032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.277654    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd" event={"ID":"aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e","Type":"ContainerDied","Data":"7485d53c5a1ad975f6100faf0295a26d95c97263ae65b3c2cb290dc2ee15c8cb"}
Nov 03 00:15:05.278032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.277886    5012 scope.go:122] "RemoveContainer" containerID="7485d53c5a1ad975f6100faf0295a26d95c97263ae65b3c2cb290dc2ee15c8cb"
Nov 03 00:15:05.279367 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.279338    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz" event={"ID":"4d57748d-abae-487e-8e34-47ff716921be","Type":"ContainerStarted","Data":"8b1a2a2237cf6ce5e431a0502ca8364c5adb98fd2bf150b21a72ffcb04622f59"}
Nov 03 00:15:05.279479 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.279385    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz" event={"ID":"4d57748d-abae-487e-8e34-47ff716921be","Type":"ContainerStarted","Data":"032a615be4e45cdef4bc5c15244e6890e750751cb080fe61f42a098ab372d7a6"}
Nov 03 00:15:05.279479 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.279472    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz"
Nov 03 00:15:05.281617 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.281595    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv" event={"ID":"ff82c8b4-dd1a-4870-829e-a37c98bf0280","Type":"ContainerStarted","Data":"fc2a04537c5769df4ea375d2cbe8cc6b887d875968f98828355b760a6aba16f9"}
Nov 03 00:15:05.281717 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.281638    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv" event={"ID":"ff82c8b4-dd1a-4870-829e-a37c98bf0280","Type":"ContainerStarted","Data":"dc7defd5d0b3d7c080f37c2e3fde352db143da9a70550326c95c4d9cf05a1116"}
Nov 03 00:15:05.281717 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.281683    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv"
Nov 03 00:15:05.367521 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.367476    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv" podStartSLOduration=2.3674633800000002 podStartE2EDuration="2.36746338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:05.364795434 +0000 UTC m=+588.244379477" watchObservedRunningTime="2026-11-03 00:15:05.36746338 +0000 UTC m=+588.247047436"
Nov 03 00:15:05.400589 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.400546    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz" podStartSLOduration=2.400533912 podStartE2EDuration="2.400533912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:05.398806082 +0000 UTC m=+588.278390126" watchObservedRunningTime="2026-11-03 00:15:05.400533912 +0000 UTC m=+588.280117952"
Nov 03 00:15:05.518515 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.518488    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff"]
Nov 03 00:15:05.523557 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.523536    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff"
Nov 03 00:15:05.539266 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.539232    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff"]
Nov 03 00:15:05.539436 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.539418    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-b4cc5943-dockercfg-x5hkg\""
Nov 03 00:15:05.539515 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.539454    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-b4cc5943\""
Nov 03 00:15:05.597890 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.597846    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76p4d\" (UniqueName: \"kubernetes.io/projected/dbf1c19b-8d46-401e-adb3-efa46f7dd7ff-kube-api-access-76p4d\") pod \"payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff\" (UID: \"dbf1c19b-8d46-401e-adb3-efa46f7dd7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff"
Nov 03 00:15:05.598115 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.597958    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbf1c19b-8d46-401e-adb3-efa46f7dd7ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff\" (UID: \"dbf1c19b-8d46-401e-adb3-efa46f7dd7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff"
Nov 03 00:15:05.699500 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.699467    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbf1c19b-8d46-401e-adb3-efa46f7dd7ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff\" (UID: \"dbf1c19b-8d46-401e-adb3-efa46f7dd7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff"
Nov 03 00:15:05.700014 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.699980    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76p4d\" (UniqueName: \"kubernetes.io/projected/dbf1c19b-8d46-401e-adb3-efa46f7dd7ff-kube-api-access-76p4d\") pod \"payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff\" (UID: \"dbf1c19b-8d46-401e-adb3-efa46f7dd7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff"
Nov 03 00:15:05.700387 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.700366    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbf1c19b-8d46-401e-adb3-efa46f7dd7ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff\" (UID: \"dbf1c19b-8d46-401e-adb3-efa46f7dd7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff"
Nov 03 00:15:05.701339 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.701312    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn"]
Nov 03 00:15:05.704716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.704696    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn"
Nov 03 00:15:05.747008 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.746983    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn"]
Nov 03 00:15:05.747731 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.747711    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76p4d\" (UniqueName: \"kubernetes.io/projected/dbf1c19b-8d46-401e-adb3-efa46f7dd7ff-kube-api-access-76p4d\") pod \"payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff\" (UID: \"dbf1c19b-8d46-401e-adb3-efa46f7dd7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff"
Nov 03 00:15:05.801437 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.801374    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6bde7c6-1acf-4517-8698-2609d2c30872-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn\" (UID: \"a6bde7c6-1acf-4517-8698-2609d2c30872\") " pod="openshift-ingress/payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn"
Nov 03 00:15:05.801437 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.801435    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2glvg\" (UniqueName: \"kubernetes.io/projected/a6bde7c6-1acf-4517-8698-2609d2c30872-kube-api-access-2glvg\") pod \"payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn\" (UID: \"a6bde7c6-1acf-4517-8698-2609d2c30872\") " pod="openshift-ingress/payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn"
Nov 03 00:15:05.836242 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.836218    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff"
Nov 03 00:15:05.911421 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.902793    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2glvg\" (UniqueName: \"kubernetes.io/projected/a6bde7c6-1acf-4517-8698-2609d2c30872-kube-api-access-2glvg\") pod \"payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn\" (UID: \"a6bde7c6-1acf-4517-8698-2609d2c30872\") " pod="openshift-ingress/payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn"
Nov 03 00:15:05.911421 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.902982    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6bde7c6-1acf-4517-8698-2609d2c30872-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn\" (UID: \"a6bde7c6-1acf-4517-8698-2609d2c30872\") " pod="openshift-ingress/payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn"
Nov 03 00:15:05.911421 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.905900    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6bde7c6-1acf-4517-8698-2609d2c30872-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn\" (UID: \"a6bde7c6-1acf-4517-8698-2609d2c30872\") " pod="openshift-ingress/payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn"
Nov 03 00:15:05.925656 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:05.925626    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2glvg\" (UniqueName: \"kubernetes.io/projected/a6bde7c6-1acf-4517-8698-2609d2c30872-kube-api-access-2glvg\") pod \"payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn\" (UID: \"a6bde7c6-1acf-4517-8698-2609d2c30872\") " pod="openshift-ingress/payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn"
Nov 03 00:15:06.020581 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.020287    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn"
Nov 03 00:15:06.148714 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.148686    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff"]
Nov 03 00:15:06.149797 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:15:06.149767    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf1c19b_8d46_401e_adb3_efa46f7dd7ff.slice/crio-109d9724bf27785ecdd649823de343702c2d3ad60b6c01692630ad4668d687a1 WatchSource:0}: Error finding container 109d9724bf27785ecdd649823de343702c2d3ad60b6c01692630ad4668d687a1: Status 404 returned error can't find the container with id 109d9724bf27785ecdd649823de343702c2d3ad60b6c01692630ad4668d687a1
Nov 03 00:15:06.289540 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.289510    5012 generic.go:356] "Generic (PLEG): container finished" podID="aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e" containerID="272529340e0b7ad1d65a900cad111c78cf4b5769e014837fdfa723f47cc9dafa" exitCode=60
Nov 03 00:15:06.289899 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.289585    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd" event={"ID":"aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e","Type":"ContainerDied","Data":"272529340e0b7ad1d65a900cad111c78cf4b5769e014837fdfa723f47cc9dafa"}
Nov 03 00:15:06.289899 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.289646    5012 scope.go:122] "RemoveContainer" containerID="7485d53c5a1ad975f6100faf0295a26d95c97263ae65b3c2cb290dc2ee15c8cb"
Nov 03 00:15:06.290036 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.289963    5012 scope.go:122] "RemoveContainer" containerID="272529340e0b7ad1d65a900cad111c78cf4b5769e014837fdfa723f47cc9dafa"
Nov 03 00:15:06.290300 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:15:06.290273    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894415-l2kjd_odh-ai-gateway-infra(aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd" podUID="aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e"
Nov 03 00:15:06.291533 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.291413    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff" event={"ID":"dbf1c19b-8d46-401e-adb3-efa46f7dd7ff","Type":"ContainerStarted","Data":"fb6357da7a8324907561b1e9bf72d236d6b0051a7d19f48b3fe9c7fc80d36fa2"}
Nov 03 00:15:06.291533 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.291459    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff" event={"ID":"dbf1c19b-8d46-401e-adb3-efa46f7dd7ff","Type":"ContainerStarted","Data":"109d9724bf27785ecdd649823de343702c2d3ad60b6c01692630ad4668d687a1"}
Nov 03 00:15:06.291701 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.291639    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff"
Nov 03 00:15:06.332623 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:15:06.332567    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6bde7c6_1acf_4517_8698_2609d2c30872.slice/crio-addd20f734a9d118f3a7ca25514c57facbe84bf995bf8af581044b68510bf9d3 WatchSource:0}: Error finding container addd20f734a9d118f3a7ca25514c57facbe84bf995bf8af581044b68510bf9d3: Status 404 returned error can't find the container with id addd20f734a9d118f3a7ca25514c57facbe84bf995bf8af581044b68510bf9d3
Nov 03 00:15:06.334258 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.334233    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn"]
Nov 03 00:15:06.376233 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.376167    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff" podStartSLOduration=1.376148805 podStartE2EDuration="1.376148805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:06.370146628 +0000 UTC m=+589.249730671" watchObservedRunningTime="2026-11-03 00:15:06.376148805 +0000 UTC m=+589.255732849"
Nov 03 00:15:06.551819 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.551787    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v"]
Nov 03 00:15:06.555785 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.555764    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v"
Nov 03 00:15:06.569434 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.569392    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a96a129b\""
Nov 03 00:15:06.569666 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.569649    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a96a129b-dockercfg-wzcn8\""
Nov 03 00:15:06.572841 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.572817    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v"]
Nov 03 00:15:06.594051 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.593987    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq"]
Nov 03 00:15:06.597901 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.597881    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq"
Nov 03 00:15:06.615353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.615335    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq"]
Nov 03 00:15:06.712968 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.712933    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvcqp\" (UniqueName: \"kubernetes.io/projected/b0147424-0774-4202-9291-da123886ff22-kube-api-access-zvcqp\") pod \"payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq\" (UID: \"b0147424-0774-4202-9291-da123886ff22\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq"
Nov 03 00:15:06.713150 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.712988    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vrxv\" (UniqueName: \"kubernetes.io/projected/f2da7b79-1d53-4f5b-a804-492511b2d75f-kube-api-access-6vrxv\") pod \"payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v\" (UID: \"f2da7b79-1d53-4f5b-a804-492511b2d75f\") " pod="openshift-ingress/payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v"
Nov 03 00:15:06.713150 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.713071    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0147424-0774-4202-9291-da123886ff22-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq\" (UID: \"b0147424-0774-4202-9291-da123886ff22\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq"
Nov 03 00:15:06.713150 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.713140    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2da7b79-1d53-4f5b-a804-492511b2d75f-plugins-config-volume\") pod \"payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v\" (UID: \"f2da7b79-1d53-4f5b-a804-492511b2d75f\") " pod="openshift-ingress/payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v"
Nov 03 00:15:06.814649 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.814620    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2da7b79-1d53-4f5b-a804-492511b2d75f-plugins-config-volume\") pod \"payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v\" (UID: \"f2da7b79-1d53-4f5b-a804-492511b2d75f\") " pod="openshift-ingress/payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v"
Nov 03 00:15:06.814811 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.814687    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvcqp\" (UniqueName: \"kubernetes.io/projected/b0147424-0774-4202-9291-da123886ff22-kube-api-access-zvcqp\") pod \"payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq\" (UID: \"b0147424-0774-4202-9291-da123886ff22\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq"
Nov 03 00:15:06.814811 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.814754    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vrxv\" (UniqueName: \"kubernetes.io/projected/f2da7b79-1d53-4f5b-a804-492511b2d75f-kube-api-access-6vrxv\") pod \"payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v\" (UID: \"f2da7b79-1d53-4f5b-a804-492511b2d75f\") " pod="openshift-ingress/payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v"
Nov 03 00:15:06.814811 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.814799    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0147424-0774-4202-9291-da123886ff22-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq\" (UID: \"b0147424-0774-4202-9291-da123886ff22\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq"
Nov 03 00:15:06.815681 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.815660    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2da7b79-1d53-4f5b-a804-492511b2d75f-plugins-config-volume\") pod \"payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v\" (UID: \"f2da7b79-1d53-4f5b-a804-492511b2d75f\") " pod="openshift-ingress/payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v"
Nov 03 00:15:06.815739 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.815657    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0147424-0774-4202-9291-da123886ff22-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq\" (UID: \"b0147424-0774-4202-9291-da123886ff22\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq"
Nov 03 00:15:06.833186 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.833160    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vrxv\" (UniqueName: \"kubernetes.io/projected/f2da7b79-1d53-4f5b-a804-492511b2d75f-kube-api-access-6vrxv\") pod \"payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v\" (UID: \"f2da7b79-1d53-4f5b-a804-492511b2d75f\") " pod="openshift-ingress/payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v"
Nov 03 00:15:06.833186 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.833175    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvcqp\" (UniqueName: \"kubernetes.io/projected/b0147424-0774-4202-9291-da123886ff22-kube-api-access-zvcqp\") pod \"payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq\" (UID: \"b0147424-0774-4202-9291-da123886ff22\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq"
Nov 03 00:15:06.869190 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.869165    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v"
Nov 03 00:15:06.912529 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:06.912500    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq"
Nov 03 00:15:07.171046 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:07.171018    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v"]
Nov 03 00:15:07.172331 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:15:07.172301    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2da7b79_1d53_4f5b_a804_492511b2d75f.slice/crio-28d9f2ed2cdd13c63e614021208800653b72224c294ed4d534441dbe7a5427c3 WatchSource:0}: Error finding container 28d9f2ed2cdd13c63e614021208800653b72224c294ed4d534441dbe7a5427c3: Status 404 returned error can't find the container with id 28d9f2ed2cdd13c63e614021208800653b72224c294ed4d534441dbe7a5427c3
Nov 03 00:15:07.248555 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:07.248529    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq"]
Nov 03 00:15:07.256031 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:15:07.256004    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0147424_0774_4202_9291_da123886ff22.slice/crio-cc545e101719af1ca66440f286b640de017fa4c6de1b506c405ca36eae84a9d6 WatchSource:0}: Error finding container cc545e101719af1ca66440f286b640de017fa4c6de1b506c405ca36eae84a9d6: Status 404 returned error can't find the container with id cc545e101719af1ca66440f286b640de017fa4c6de1b506c405ca36eae84a9d6
Nov 03 00:15:07.325490 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:07.325460    5012 scope.go:122] "RemoveContainer" containerID="272529340e0b7ad1d65a900cad111c78cf4b5769e014837fdfa723f47cc9dafa"
Nov 03 00:15:07.325831 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:15:07.325681    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894415-l2kjd_odh-ai-gateway-infra(aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd" podUID="aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e"
Nov 03 00:15:07.326579 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:07.326555    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn" event={"ID":"a6bde7c6-1acf-4517-8698-2609d2c30872","Type":"ContainerStarted","Data":"5d13350bc6ce81a8159d259bb0d288328d7fe4a3a7bcbf71fd2a539590b17fbd"}
Nov 03 00:15:07.326628 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:07.326600    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn" event={"ID":"a6bde7c6-1acf-4517-8698-2609d2c30872","Type":"ContainerStarted","Data":"addd20f734a9d118f3a7ca25514c57facbe84bf995bf8af581044b68510bf9d3"}
Nov 03 00:15:07.326662 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:07.326634    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn"
Nov 03 00:15:07.327893 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:07.327878    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq" event={"ID":"b0147424-0774-4202-9291-da123886ff22","Type":"ContainerStarted","Data":"cc545e101719af1ca66440f286b640de017fa4c6de1b506c405ca36eae84a9d6"}
Nov 03 00:15:07.329269 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:07.329247    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v" event={"ID":"f2da7b79-1d53-4f5b-a804-492511b2d75f","Type":"ContainerStarted","Data":"455a0b7def04c809eb881d51a4f767e8fd433668cbb5fd14654189965eb9e324"}
Nov 03 00:15:07.329372 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:07.329285    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v" event={"ID":"f2da7b79-1d53-4f5b-a804-492511b2d75f","Type":"ContainerStarted","Data":"28d9f2ed2cdd13c63e614021208800653b72224c294ed4d534441dbe7a5427c3"}
Nov 03 00:15:07.329420 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:07.329382    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v"
Nov 03 00:15:07.364939 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:15:07.364895    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:15:07.448449 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:07.448342    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v" podStartSLOduration=1.44832796 podStartE2EDuration="1.44832796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:07.446754864 +0000 UTC m=+590.326338907" watchObservedRunningTime="2026-11-03 00:15:07.44832796 +0000 UTC m=+590.327912006"
Nov 03 00:15:07.450205 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:07.450159    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn" podStartSLOduration=2.450147497 podStartE2EDuration="2.450147497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:07.412206854 +0000 UTC m=+590.291790898" watchObservedRunningTime="2026-11-03 00:15:07.450147497 +0000 UTC m=+590.329731544"
Nov 03 00:15:08.022575 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.022541    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"]
Nov 03 00:15:08.027190 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.027168    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.052922 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.052884    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-e3e6ed37-openshift-default-dockercfg-nsndp\""
Nov 03 00:15:08.064987 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.064956    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"]
Nov 03 00:15:08.129442 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.129401    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-workload-socket\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.129636 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.129562    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-workload-certs\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.129636 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.129615    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e2d4e66-888d-455b-bded-5db746148762-istio-token\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.129777 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.129650    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-credential-socket\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.129777 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.129677    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-istio-envoy\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.129777 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.129705    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e2d4e66-888d-455b-bded-5db746148762-istiod-ca-cert\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.129777 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.129728    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlxkj\" (UniqueName: \"kubernetes.io/projected/4e2d4e66-888d-455b-bded-5db746148762-kube-api-access-dlxkj\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.129777 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.129769    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e2d4e66-888d-455b-bded-5db746148762-istio-podinfo\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.129979 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.129822    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-istio-data\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.230578 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.230540    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-workload-certs\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.230578 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.230582    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e2d4e66-888d-455b-bded-5db746148762-istio-token\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.230776 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.230601    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-credential-socket\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.230776 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.230620    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-istio-envoy\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.230776 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.230739    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e2d4e66-888d-455b-bded-5db746148762-istiod-ca-cert\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.230776 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.230769    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlxkj\" (UniqueName: \"kubernetes.io/projected/4e2d4e66-888d-455b-bded-5db746148762-kube-api-access-dlxkj\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.231000 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.230804    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e2d4e66-888d-455b-bded-5db746148762-istio-podinfo\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.231000 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.230853    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-istio-data\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.231000 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.230954    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-workload-socket\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.231152 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.231007    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-workload-certs\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.231152 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.231026    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-credential-socket\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.231270 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.231238    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-workload-socket\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.231346 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.231309    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-istio-data\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.231508 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.231485    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e2d4e66-888d-455b-bded-5db746148762-istiod-ca-cert\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.232972 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.232952    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-istio-envoy\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.233172 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.233154    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e2d4e66-888d-455b-bded-5db746148762-istio-podinfo\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.247190 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.247167    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e2d4e66-888d-455b-bded-5db746148762-istio-token\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.248295 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.248197    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlxkj\" (UniqueName: \"kubernetes.io/projected/4e2d4e66-888d-455b-bded-5db746148762-kube-api-access-dlxkj\") pod \"e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") " pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.336867 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.336777    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq" event={"ID":"b0147424-0774-4202-9291-da123886ff22","Type":"ContainerStarted","Data":"5265cb148b3035bca37b6f805ad5514a24f3aecd7c755ee8f361391a711d7ce6"}
Nov 03 00:15:08.337340 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.337074    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq"
Nov 03 00:15:08.348397 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.348367    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:08.370730 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.370688    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq" podStartSLOduration=2.370676176 podStartE2EDuration="2.370676176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:08.369765788 +0000 UTC m=+591.249349830" watchObservedRunningTime="2026-11-03 00:15:08.370676176 +0000 UTC m=+591.250260219"
Nov 03 00:15:08.684960 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.684931    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"]
Nov 03 00:15:08.685641 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:15:08.685615    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e2d4e66_888d_455b_bded_5db746148762.slice/crio-e82c30b837265791892392a42510332e46491af8f74410b84352cbb862423fed WatchSource:0}: Error finding container e82c30b837265791892392a42510332e46491af8f74410b84352cbb862423fed: Status 404 returned error can't find the container with id e82c30b837265791892392a42510332e46491af8f74410b84352cbb862423fed
Nov 03 00:15:08.713249 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.713218    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:15:08.713342 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.713283    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:15:08.713342 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.713313    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:15:08.734894 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.734872    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq"]
Nov 03 00:15:08.739882 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.739866    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq"
Nov 03 00:15:08.754738 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.754508    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq"]
Nov 03 00:15:08.755494 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.755470    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-19590f89\""
Nov 03 00:15:08.755697 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.755673    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-19590f89-dockercfg-fx6wm\""
Nov 03 00:15:08.844763 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.844498    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whnfm\" (UniqueName: \"kubernetes.io/projected/6443af0a-ce37-4412-b8e2-97d5a545ba52-kube-api-access-whnfm\") pod \"payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq\" (UID: \"6443af0a-ce37-4412-b8e2-97d5a545ba52\") " pod="openshift-ingress/payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq"
Nov 03 00:15:08.844763 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.844592    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6443af0a-ce37-4412-b8e2-97d5a545ba52-plugins-config-volume\") pod \"payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq\" (UID: \"6443af0a-ce37-4412-b8e2-97d5a545ba52\") " pod="openshift-ingress/payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq"
Nov 03 00:15:08.946021 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.945940    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6443af0a-ce37-4412-b8e2-97d5a545ba52-plugins-config-volume\") pod \"payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq\" (UID: \"6443af0a-ce37-4412-b8e2-97d5a545ba52\") " pod="openshift-ingress/payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq"
Nov 03 00:15:08.946160 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.946104    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whnfm\" (UniqueName: \"kubernetes.io/projected/6443af0a-ce37-4412-b8e2-97d5a545ba52-kube-api-access-whnfm\") pod \"payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq\" (UID: \"6443af0a-ce37-4412-b8e2-97d5a545ba52\") " pod="openshift-ingress/payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq"
Nov 03 00:15:08.946779 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.946759    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6443af0a-ce37-4412-b8e2-97d5a545ba52-plugins-config-volume\") pod \"payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq\" (UID: \"6443af0a-ce37-4412-b8e2-97d5a545ba52\") " pod="openshift-ingress/payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq"
Nov 03 00:15:08.962939 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:08.962898    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whnfm\" (UniqueName: \"kubernetes.io/projected/6443af0a-ce37-4412-b8e2-97d5a545ba52-kube-api-access-whnfm\") pod \"payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq\" (UID: \"6443af0a-ce37-4412-b8e2-97d5a545ba52\") " pod="openshift-ingress/payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq"
Nov 03 00:15:09.064164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:09.064133    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq"
Nov 03 00:15:09.345426 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:09.345336    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z" event={"ID":"4e2d4e66-888d-455b-bded-5db746148762","Type":"ContainerStarted","Data":"9536692556bfaba6cb0d5ba03d940dc98589f328a3b7ea8ecd8317591603de30"}
Nov 03 00:15:09.345426 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:09.345389    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z" event={"ID":"4e2d4e66-888d-455b-bded-5db746148762","Type":"ContainerStarted","Data":"e82c30b837265791892392a42510332e46491af8f74410b84352cbb862423fed"}
Nov 03 00:15:09.348799 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:09.348774    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:09.354177 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:09.354151    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:09.381928 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:09.381852    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z" podStartSLOduration=2.381836997 podStartE2EDuration="2.381836997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:09.380545035 +0000 UTC m=+592.260129077" watchObservedRunningTime="2026-11-03 00:15:09.381836997 +0000 UTC m=+592.261421041"
Nov 03 00:15:09.412068 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:09.411016    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq"]
Nov 03 00:15:09.415987 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:15:09.415955    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6443af0a_ce37_4412_b8e2_97d5a545ba52.slice/crio-53df2c50e6328f19b9483a265959c7d9a42d586505faacfb78e0e415562fd983 WatchSource:0}: Error finding container 53df2c50e6328f19b9483a265959c7d9a42d586505faacfb78e0e415562fd983: Status 404 returned error can't find the container with id 53df2c50e6328f19b9483a265959c7d9a42d586505faacfb78e0e415562fd983
Nov 03 00:15:10.370931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:10.368495    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq" event={"ID":"6443af0a-ce37-4412-b8e2-97d5a545ba52","Type":"ContainerStarted","Data":"22e0c03014d4d9514fedb54be888b534b452b7030848ef533835d7ef5a50824b"}
Nov 03 00:15:10.370931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:10.368581    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq" event={"ID":"6443af0a-ce37-4412-b8e2-97d5a545ba52","Type":"ContainerStarted","Data":"53df2c50e6328f19b9483a265959c7d9a42d586505faacfb78e0e415562fd983"}
Nov 03 00:15:10.370931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:10.368657    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq"
Nov 03 00:15:10.370931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:10.369359    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:10.370931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:10.370527    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:10.539114 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:10.537212    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq" podStartSLOduration=2.537189069 podStartE2EDuration="2.537189069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:10.485482338 +0000 UTC m=+593.365066393" watchObservedRunningTime="2026-11-03 00:15:10.537189069 +0000 UTC m=+593.416773113"
Nov 03 00:15:13.055931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:13.055831    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2"]
Nov 03 00:15:13.061360 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:13.061328    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2"
Nov 03 00:15:13.075949 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:13.075895    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2"]
Nov 03 00:15:13.082228 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:13.082201    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-e3e6ed37\""
Nov 03 00:15:13.082371 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:13.082207    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-e3e6ed37-dockercfg-cx54q\""
Nov 03 00:15:13.255809 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:13.255774    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a168116a-4202-4d6d-9007-e54f691c1a76-plugins-config-volume\") pod \"payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2\" (UID: \"a168116a-4202-4d6d-9007-e54f691c1a76\") " pod="openshift-ingress/payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2"
Nov 03 00:15:13.255994 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:13.255922    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2wq6\" (UniqueName: \"kubernetes.io/projected/a168116a-4202-4d6d-9007-e54f691c1a76-kube-api-access-r2wq6\") pod \"payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2\" (UID: \"a168116a-4202-4d6d-9007-e54f691c1a76\") " pod="openshift-ingress/payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2"
Nov 03 00:15:13.356927 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:13.356885    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a168116a-4202-4d6d-9007-e54f691c1a76-plugins-config-volume\") pod \"payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2\" (UID: \"a168116a-4202-4d6d-9007-e54f691c1a76\") " pod="openshift-ingress/payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2"
Nov 03 00:15:13.357055 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:13.356989    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2wq6\" (UniqueName: \"kubernetes.io/projected/a168116a-4202-4d6d-9007-e54f691c1a76-kube-api-access-r2wq6\") pod \"payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2\" (UID: \"a168116a-4202-4d6d-9007-e54f691c1a76\") " pod="openshift-ingress/payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2"
Nov 03 00:15:13.357641 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:13.357624    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a168116a-4202-4d6d-9007-e54f691c1a76-plugins-config-volume\") pod \"payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2\" (UID: \"a168116a-4202-4d6d-9007-e54f691c1a76\") " pod="openshift-ingress/payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2"
Nov 03 00:15:13.380241 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:13.380212    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2wq6\" (UniqueName: \"kubernetes.io/projected/a168116a-4202-4d6d-9007-e54f691c1a76-kube-api-access-r2wq6\") pod \"payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2\" (UID: \"a168116a-4202-4d6d-9007-e54f691c1a76\") " pod="openshift-ingress/payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2"
Nov 03 00:15:13.677443 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:13.677359    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2"
Nov 03 00:15:14.017814 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:15:14.017780    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda168116a_4202_4d6d_9007_e54f691c1a76.slice/crio-cdecfc09ccd7cba7c828d28fa0d478bd47259c8b1eedb2edef12347fdb206076 WatchSource:0}: Error finding container cdecfc09ccd7cba7c828d28fa0d478bd47259c8b1eedb2edef12347fdb206076: Status 404 returned error can't find the container with id cdecfc09ccd7cba7c828d28fa0d478bd47259c8b1eedb2edef12347fdb206076
Nov 03 00:15:14.035930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:14.026744    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2"]
Nov 03 00:15:14.428795 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:14.428752    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2" event={"ID":"a168116a-4202-4d6d-9007-e54f691c1a76","Type":"ContainerStarted","Data":"6c441df31afb23373b4b8ab4a0042680c912c7a2bf99b1576509077f21d2893a"}
Nov 03 00:15:14.429250 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:14.428817    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2" event={"ID":"a168116a-4202-4d6d-9007-e54f691c1a76","Type":"ContainerStarted","Data":"cdecfc09ccd7cba7c828d28fa0d478bd47259c8b1eedb2edef12347fdb206076"}
Nov 03 00:15:14.429250 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:14.429010    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2"
Nov 03 00:15:14.465091 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:14.465037    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2" podStartSLOduration=1.465024563 podStartE2EDuration="1.465024563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:14.461809139 +0000 UTC m=+597.341393183" watchObservedRunningTime="2026-11-03 00:15:14.465024563 +0000 UTC m=+597.344608648"
Nov 03 00:15:16.294374 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:16.294299    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz"
Nov 03 00:15:16.294374 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:16.294364    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv"
Nov 03 00:15:16.807287 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:16.803721    5012 patch_prober.go:29] interesting pod/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": read tcp 10.132.0.2:49058->10.132.0.61:15021: read: connection reset by peer" start-of-body=
Nov 03 00:15:16.807287 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:16.803777    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z" podUID="c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": read tcp 10.132.0.2:49058->10.132.0.61:15021: read: connection reset by peer"
Nov 03 00:15:17.338233 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:17.338139    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff"
Nov 03 00:15:17.361581 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:17.361562    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 00:15:17.361722 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:17.361710    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 00:15:17.367043 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:17.367020    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:15:17.367617 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:17.367598    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:15:17.452637 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:17.452606    5012 generic.go:356] "Generic (PLEG): container finished" podID="c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" containerID="1cb8593e890e50a46b1b1df8ece05acf229d7e7740908ba472b6785a215be084" exitCode=0
Nov 03 00:15:17.452820 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:17.452670    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z" event={"ID":"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8","Type":"ContainerDied","Data":"1cb8593e890e50a46b1b1df8ece05acf229d7e7740908ba472b6785a215be084"}
Nov 03 00:15:17.453151 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:17.453130    5012 scope.go:122] "RemoveContainer" containerID="1cb8593e890e50a46b1b1df8ece05acf229d7e7740908ba472b6785a215be084"
Nov 03 00:15:17.454594 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:17.454563    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:15:17.454662 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:17.454637    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:15:17.454699 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:17.454667    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:15:18.236327 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:18.236298    5012 scope.go:122] "RemoveContainer" containerID="272529340e0b7ad1d65a900cad111c78cf4b5769e014837fdfa723f47cc9dafa"
Nov 03 00:15:18.316942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:18.313023    5012 patch_prober.go:29] interesting pod/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:15:18.316942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:18.313086    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv" podUID="97953a4d-b832-4282-b010-d06cc1e7e3c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:15:18.339678 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:18.339652    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v"
Nov 03 00:15:18.340072 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:18.339722    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn"
Nov 03 00:15:18.462344 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:18.462306    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z" event={"ID":"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8","Type":"ContainerStarted","Data":"65d27ca51e212ea86cbf9b7e19f636692a4d0898f2d6398667e7850ecc96e586"}
Nov 03 00:15:18.609932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:18.604955    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:15:18.609932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:18.605100    5012 patch_prober.go:29] interesting pod/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 00:15:18.609932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:18.605162    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z" podUID="c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 00:15:19.349261 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:19.349232    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq"
Nov 03 00:15:19.473138 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:19.473064    5012 generic.go:356] "Generic (PLEG): container finished" podID="aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e" containerID="467e79f9e55ff0c1b28858b43261cab4adbb32500fcabe0618b8860e162b4135" exitCode=60
Nov 03 00:15:19.473279 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:19.473140    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd" event={"ID":"aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e","Type":"ContainerDied","Data":"467e79f9e55ff0c1b28858b43261cab4adbb32500fcabe0618b8860e162b4135"}
Nov 03 00:15:19.473279 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:19.473244    5012 scope.go:122] "RemoveContainer" containerID="272529340e0b7ad1d65a900cad111c78cf4b5769e014837fdfa723f47cc9dafa"
Nov 03 00:15:19.473613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:19.473575    5012 scope.go:122] "RemoveContainer" containerID="467e79f9e55ff0c1b28858b43261cab4adbb32500fcabe0618b8860e162b4135"
Nov 03 00:15:19.473866 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:15:19.473837    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894415-l2kjd_odh-ai-gateway-infra(aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd" podUID="aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e"
Nov 03 00:15:19.475254 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:19.475234    5012 generic.go:356] "Generic (PLEG): container finished" podID="97953a4d-b832-4282-b010-d06cc1e7e3c2" containerID="6af6579a784d160b63a9ac72aabf7df7d1992c6e4a6a3505ce8607ed34d68cea" exitCode=0
Nov 03 00:15:19.475366 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:19.475306    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv" event={"ID":"97953a4d-b832-4282-b010-d06cc1e7e3c2","Type":"ContainerDied","Data":"6af6579a784d160b63a9ac72aabf7df7d1992c6e4a6a3505ce8607ed34d68cea"}
Nov 03 00:15:19.475784 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:19.475766    5012 scope.go:122] "RemoveContainer" containerID="6af6579a784d160b63a9ac72aabf7df7d1992c6e4a6a3505ce8607ed34d68cea"
Nov 03 00:15:19.476861 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:19.476835    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:15:19.476946 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:19.476894    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:15:19.476992 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:19.476955    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:15:19.616934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:19.616710    5012 patch_prober.go:29] interesting pod/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 00:15:19.616934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:19.616767    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z" podUID="c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 00:15:20.485788 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:20.485753    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv" event={"ID":"97953a4d-b832-4282-b010-d06cc1e7e3c2","Type":"ContainerStarted","Data":"63559afcdbc761b278af18d1a986f386e72ea2cbad2ff7005922d94fa42f80f3"}
Nov 03 00:15:20.546127 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:20.546097    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd"]
Nov 03 00:15:20.596127 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:20.594293    5012 patch_prober.go:29] interesting pod/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 00:15:20.596127 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:20.594352    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z" podUID="c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 00:15:20.853756 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:20.853515    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd"
Nov 03 00:15:20.965560 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:20.965525    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e-maas-api-service-ca\") pod \"aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e\" (UID: \"aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e\") "
Nov 03 00:15:20.965749 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:20.965624    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e-kube-api-access-f74wc\" (UniqueName: \"kubernetes.io/projected/aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e-kube-api-access-f74wc\") pod \"aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e\" (UID: \"aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e\") "
Nov 03 00:15:20.966232 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:20.966203    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e-maas-api-service-ca" pod "aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e" (UID: "aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:20.970297 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:20.969369    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e-kube-api-access-f74wc" pod "aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e" (UID: "aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e"). InnerVolumeSpecName "kube-api-access-f74wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:21.057271 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:21.057120    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:15:21.057271 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:21.057147    5012 patch_prober.go:29] interesting pod/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:15:21.057271 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:21.057186    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv" podUID="97953a4d-b832-4282-b010-d06cc1e7e3c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 03 00:15:21.067345 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:21.067318    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e-maas-api-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:21.067345 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:21.067343    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f74wc\" (UniqueName: \"kubernetes.io/projected/aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e-kube-api-access-f74wc\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:21.377922 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:21.377878    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq"
Nov 03 00:15:21.543179 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:21.543056    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd" event={"ID":"aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e","Type":"ContainerDied","Data":"67bcadff723ddcb7ea7705e1d08d5e1c8ad0b71809a4d34c52b8f30e125a6d98"}
Nov 03 00:15:21.543179 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:21.543123    5012 scope.go:122] "RemoveContainer" containerID="467e79f9e55ff0c1b28858b43261cab4adbb32500fcabe0618b8860e162b4135"
Nov 03 00:15:21.543609 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:21.543196    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd"
Nov 03 00:15:21.618933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:21.607514    5012 patch_prober.go:29] interesting pod/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 00:15:21.618933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:21.607566    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z" podUID="c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 00:15:21.618933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:21.607866    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd"]
Nov 03 00:15:21.639989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:21.623802    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894415-l2kjd"]
Nov 03 00:15:22.056455 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:22.056380    5012 patch_prober.go:29] interesting pod/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:15:22.056455 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:22.056431    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv" podUID="97953a4d-b832-4282-b010-d06cc1e7e3c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 03 00:15:22.602936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:22.599374    5012 patch_prober.go:29] interesting pod/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 00:15:22.602936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:22.599439    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z" podUID="c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 00:15:23.056691 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:23.056609    5012 patch_prober.go:29] interesting pod/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:15:23.056691 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:23.056666    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv" podUID="97953a4d-b832-4282-b010-d06cc1e7e3c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 03 00:15:23.243663 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:23.243626    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e" path="/var/lib/kubelet/pods/aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e/volumes"
Nov 03 00:15:23.597935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:23.595336    5012 patch_prober.go:29] interesting pod/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 00:15:23.597935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:23.595400    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z" podUID="c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 00:15:24.057176 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:24.057094    5012 patch_prober.go:29] interesting pod/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:15:24.057176 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:24.057148    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv" podUID="97953a4d-b832-4282-b010-d06cc1e7e3c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 03 00:15:24.614222 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:24.614189    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:15:24.614603 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:24.614577    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:15:24.615273 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:24.615250    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:15:25.057445 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:25.057371    5012 patch_prober.go:29] interesting pod/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:15:25.057445 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:25.057426    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv" podUID="97953a4d-b832-4282-b010-d06cc1e7e3c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 03 00:15:25.438103 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:25.438076    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2"
Nov 03 00:15:26.060859 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:26.060829    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:15:26.061296 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:26.061131    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:15:26.061793 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:26.061778    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:15:28.454788 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:28.454749    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn"]
Nov 03 00:15:28.484190 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:28.484166    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff"]
Nov 03 00:15:29.781953 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:29.781892    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv"]
Nov 03 00:15:29.819502 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:29.819468    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz"]
Nov 03 00:15:33.455872 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:33.455816    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn" podUID="a6bde7c6-1acf-4517-8698-2609d2c30872" containerName="payload-processing" containerID="cri-o://5d13350bc6ce81a8159d259bb0d288328d7fe4a3a7bcbf71fd2a539590b17fbd" gracePeriod=25
Nov 03 00:15:33.485307 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:33.485269    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff" podUID="dbf1c19b-8d46-401e-adb3-efa46f7dd7ff" containerName="payload-pre-processing" containerID="cri-o://fb6357da7a8324907561b1e9bf72d236d6b0051a7d19f48b3fe9c7fc80d36fa2" gracePeriod=25
Nov 03 00:15:33.664418 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:33.664381    5012 generic.go:356] "Generic (PLEG): container finished" podID="a6bde7c6-1acf-4517-8698-2609d2c30872" containerID="5d13350bc6ce81a8159d259bb0d288328d7fe4a3a7bcbf71fd2a539590b17fbd" exitCode=0
Nov 03 00:15:33.664632 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:33.664482    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn" event={"ID":"a6bde7c6-1acf-4517-8698-2609d2c30872","Type":"ContainerDied","Data":"5d13350bc6ce81a8159d259bb0d288328d7fe4a3a7bcbf71fd2a539590b17fbd"}
Nov 03 00:15:33.666689 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:33.666661    5012 generic.go:356] "Generic (PLEG): container finished" podID="dbf1c19b-8d46-401e-adb3-efa46f7dd7ff" containerID="fb6357da7a8324907561b1e9bf72d236d6b0051a7d19f48b3fe9c7fc80d36fa2" exitCode=0
Nov 03 00:15:33.666819 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:33.666731    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff" event={"ID":"dbf1c19b-8d46-401e-adb3-efa46f7dd7ff","Type":"ContainerDied","Data":"fb6357da7a8324907561b1e9bf72d236d6b0051a7d19f48b3fe9c7fc80d36fa2"}
Nov 03 00:15:33.744883 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:33.744853    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn"
Nov 03 00:15:33.777368 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:33.777344    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff"
Nov 03 00:15:33.864206 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:33.864171    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6bde7c6-1acf-4517-8698-2609d2c30872-kube-api-access-2glvg\" (UniqueName: \"kubernetes.io/projected/a6bde7c6-1acf-4517-8698-2609d2c30872-kube-api-access-2glvg\") pod \"a6bde7c6-1acf-4517-8698-2609d2c30872\" (UID: \"a6bde7c6-1acf-4517-8698-2609d2c30872\") "
Nov 03 00:15:33.864400 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:33.864239    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6bde7c6-1acf-4517-8698-2609d2c30872-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6bde7c6-1acf-4517-8698-2609d2c30872-plugins-config-volume\") pod \"a6bde7c6-1acf-4517-8698-2609d2c30872\" (UID: \"a6bde7c6-1acf-4517-8698-2609d2c30872\") "
Nov 03 00:15:33.864400 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:33.864371    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbf1c19b-8d46-401e-adb3-efa46f7dd7ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbf1c19b-8d46-401e-adb3-efa46f7dd7ff-plugins-config-volume\") pod \"dbf1c19b-8d46-401e-adb3-efa46f7dd7ff\" (UID: \"dbf1c19b-8d46-401e-adb3-efa46f7dd7ff\") "
Nov 03 00:15:33.865405 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:33.865372    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6bde7c6-1acf-4517-8698-2609d2c30872-plugins-config-volume" pod "a6bde7c6-1acf-4517-8698-2609d2c30872" (UID: "a6bde7c6-1acf-4517-8698-2609d2c30872"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:33.865888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:33.865861    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbf1c19b-8d46-401e-adb3-efa46f7dd7ff-plugins-config-volume" pod "dbf1c19b-8d46-401e-adb3-efa46f7dd7ff" (UID: "dbf1c19b-8d46-401e-adb3-efa46f7dd7ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:33.866213 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:33.866191    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbf1c19b-8d46-401e-adb3-efa46f7dd7ff-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:33.866290 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:33.866218    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6bde7c6-1acf-4517-8698-2609d2c30872-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:33.876361 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:33.876332    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6bde7c6-1acf-4517-8698-2609d2c30872-kube-api-access-2glvg" pod "a6bde7c6-1acf-4517-8698-2609d2c30872" (UID: "a6bde7c6-1acf-4517-8698-2609d2c30872"). InnerVolumeSpecName "kube-api-access-2glvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:33.967220 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:33.967158    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbf1c19b-8d46-401e-adb3-efa46f7dd7ff-kube-api-access-76p4d\" (UniqueName: \"kubernetes.io/projected/dbf1c19b-8d46-401e-adb3-efa46f7dd7ff-kube-api-access-76p4d\") pod \"dbf1c19b-8d46-401e-adb3-efa46f7dd7ff\" (UID: \"dbf1c19b-8d46-401e-adb3-efa46f7dd7ff\") "
Nov 03 00:15:33.967380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:33.967367    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2glvg\" (UniqueName: \"kubernetes.io/projected/a6bde7c6-1acf-4517-8698-2609d2c30872-kube-api-access-2glvg\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:33.969089 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:33.969060    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf1c19b-8d46-401e-adb3-efa46f7dd7ff-kube-api-access-76p4d" pod "dbf1c19b-8d46-401e-adb3-efa46f7dd7ff" (UID: "dbf1c19b-8d46-401e-adb3-efa46f7dd7ff"). InnerVolumeSpecName "kube-api-access-76p4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:34.068409 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:34.068377    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76p4d\" (UniqueName: \"kubernetes.io/projected/dbf1c19b-8d46-401e-adb3-efa46f7dd7ff-kube-api-access-76p4d\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:34.674955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:34.674901    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn"
Nov 03 00:15:34.675422 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:34.674901    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn" event={"ID":"a6bde7c6-1acf-4517-8698-2609d2c30872","Type":"ContainerDied","Data":"addd20f734a9d118f3a7ca25514c57facbe84bf995bf8af581044b68510bf9d3"}
Nov 03 00:15:34.675422 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:34.675111    5012 scope.go:122] "RemoveContainer" containerID="5d13350bc6ce81a8159d259bb0d288328d7fe4a3a7bcbf71fd2a539590b17fbd"
Nov 03 00:15:34.676675 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:34.676623    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff" event={"ID":"dbf1c19b-8d46-401e-adb3-efa46f7dd7ff","Type":"ContainerDied","Data":"109d9724bf27785ecdd649823de343702c2d3ad60b6c01692630ad4668d687a1"}
Nov 03 00:15:34.676754 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:34.676727    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff"
Nov 03 00:15:34.688516 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:34.688497    5012 scope.go:122] "RemoveContainer" containerID="fb6357da7a8324907561b1e9bf72d236d6b0051a7d19f48b3fe9c7fc80d36fa2"
Nov 03 00:15:34.733297 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:34.733236    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn"]
Nov 03 00:15:34.737163 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:34.737145    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b4cc5943-867dbdb575-xqjtn"]
Nov 03 00:15:34.765309 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:34.765287    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff"]
Nov 03 00:15:34.769782 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:34.769762    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b4cc5943-854759774c-7h5ff"]
Nov 03 00:15:34.782935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:34.782879    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv" podUID="ff82c8b4-dd1a-4870-829e-a37c98bf0280" containerName="payload-processing" containerID="cri-o://fc2a04537c5769df4ea375d2cbe8cc6b887d875968f98828355b760a6aba16f9" gracePeriod=25
Nov 03 00:15:34.821993 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:34.820562    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz" podUID="4d57748d-abae-487e-8e34-47ff716921be" containerName="payload-pre-processing" containerID="cri-o://8b1a2a2237cf6ce5e431a0502ca8364c5adb98fd2bf150b21a72ffcb04622f59" gracePeriod=25
Nov 03 00:15:35.075030 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.074983    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv"
Nov 03 00:15:35.102660 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.102640    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz"
Nov 03 00:15:35.180776 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.180750    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff82c8b4-dd1a-4870-829e-a37c98bf0280-kube-api-access-xr6hf\" (UniqueName: \"kubernetes.io/projected/ff82c8b4-dd1a-4870-829e-a37c98bf0280-kube-api-access-xr6hf\") pod \"ff82c8b4-dd1a-4870-829e-a37c98bf0280\" (UID: \"ff82c8b4-dd1a-4870-829e-a37c98bf0280\") "
Nov 03 00:15:35.180925 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.180850    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff82c8b4-dd1a-4870-829e-a37c98bf0280-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff82c8b4-dd1a-4870-829e-a37c98bf0280-plugins-config-volume\") pod \"ff82c8b4-dd1a-4870-829e-a37c98bf0280\" (UID: \"ff82c8b4-dd1a-4870-829e-a37c98bf0280\") "
Nov 03 00:15:35.181229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.181204    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff82c8b4-dd1a-4870-829e-a37c98bf0280-plugins-config-volume" pod "ff82c8b4-dd1a-4870-829e-a37c98bf0280" (UID: "ff82c8b4-dd1a-4870-829e-a37c98bf0280"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:35.182777 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.182757    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff82c8b4-dd1a-4870-829e-a37c98bf0280-kube-api-access-xr6hf" pod "ff82c8b4-dd1a-4870-829e-a37c98bf0280" (UID: "ff82c8b4-dd1a-4870-829e-a37c98bf0280"). InnerVolumeSpecName "kube-api-access-xr6hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:35.243028 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.243000    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6bde7c6-1acf-4517-8698-2609d2c30872" path="/var/lib/kubelet/pods/a6bde7c6-1acf-4517-8698-2609d2c30872/volumes"
Nov 03 00:15:35.243374 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.243361    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbf1c19b-8d46-401e-adb3-efa46f7dd7ff" path="/var/lib/kubelet/pods/dbf1c19b-8d46-401e-adb3-efa46f7dd7ff/volumes"
Nov 03 00:15:35.267272 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.267251    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq"]
Nov 03 00:15:35.281944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.281921    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d57748d-abae-487e-8e34-47ff716921be-kube-api-access-rltgd\" (UniqueName: \"kubernetes.io/projected/4d57748d-abae-487e-8e34-47ff716921be-kube-api-access-rltgd\") pod \"4d57748d-abae-487e-8e34-47ff716921be\" (UID: \"4d57748d-abae-487e-8e34-47ff716921be\") "
Nov 03 00:15:35.282054 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.281952    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d57748d-abae-487e-8e34-47ff716921be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d57748d-abae-487e-8e34-47ff716921be-plugins-config-volume\") pod \"4d57748d-abae-487e-8e34-47ff716921be\" (UID: \"4d57748d-abae-487e-8e34-47ff716921be\") "
Nov 03 00:15:35.282199 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.282185    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff82c8b4-dd1a-4870-829e-a37c98bf0280-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:35.282267 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.282200    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr6hf\" (UniqueName: \"kubernetes.io/projected/ff82c8b4-dd1a-4870-829e-a37c98bf0280-kube-api-access-xr6hf\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:35.282324 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.282305    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d57748d-abae-487e-8e34-47ff716921be-plugins-config-volume" pod "4d57748d-abae-487e-8e34-47ff716921be" (UID: "4d57748d-abae-487e-8e34-47ff716921be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:35.283845 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.283824    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d57748d-abae-487e-8e34-47ff716921be-kube-api-access-rltgd" pod "4d57748d-abae-487e-8e34-47ff716921be" (UID: "4d57748d-abae-487e-8e34-47ff716921be"). InnerVolumeSpecName "kube-api-access-rltgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:35.383140 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.383118    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rltgd\" (UniqueName: \"kubernetes.io/projected/4d57748d-abae-487e-8e34-47ff716921be-kube-api-access-rltgd\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:35.383140 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.383141    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d57748d-abae-487e-8e34-47ff716921be-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:35.684532 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.684455    5012 generic.go:356] "Generic (PLEG): container finished" podID="4d57748d-abae-487e-8e34-47ff716921be" containerID="8b1a2a2237cf6ce5e431a0502ca8364c5adb98fd2bf150b21a72ffcb04622f59" exitCode=0
Nov 03 00:15:35.684532 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.684495    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz"
Nov 03 00:15:35.685030 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.684534    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz" event={"ID":"4d57748d-abae-487e-8e34-47ff716921be","Type":"ContainerDied","Data":"8b1a2a2237cf6ce5e431a0502ca8364c5adb98fd2bf150b21a72ffcb04622f59"}
Nov 03 00:15:35.685030 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.684582    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz" event={"ID":"4d57748d-abae-487e-8e34-47ff716921be","Type":"ContainerDied","Data":"032a615be4e45cdef4bc5c15244e6890e750751cb080fe61f42a098ab372d7a6"}
Nov 03 00:15:35.685030 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.684611    5012 scope.go:122] "RemoveContainer" containerID="8b1a2a2237cf6ce5e431a0502ca8364c5adb98fd2bf150b21a72ffcb04622f59"
Nov 03 00:15:35.686106 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.686079    5012 generic.go:356] "Generic (PLEG): container finished" podID="ff82c8b4-dd1a-4870-829e-a37c98bf0280" containerID="fc2a04537c5769df4ea375d2cbe8cc6b887d875968f98828355b760a6aba16f9" exitCode=0
Nov 03 00:15:35.686222 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.686117    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv"
Nov 03 00:15:35.686222 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.686157    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv" event={"ID":"ff82c8b4-dd1a-4870-829e-a37c98bf0280","Type":"ContainerDied","Data":"fc2a04537c5769df4ea375d2cbe8cc6b887d875968f98828355b760a6aba16f9"}
Nov 03 00:15:35.686222 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.686190    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv" event={"ID":"ff82c8b4-dd1a-4870-829e-a37c98bf0280","Type":"ContainerDied","Data":"dc7defd5d0b3d7c080f37c2e3fde352db143da9a70550326c95c4d9cf05a1116"}
Nov 03 00:15:35.690040 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.690017    5012 generic.go:356] "Generic (PLEG): container finished" podID="4e2d4e66-888d-455b-bded-5db746148762" containerID="9536692556bfaba6cb0d5ba03d940dc98589f328a3b7ea8ecd8317591603de30" exitCode=0
Nov 03 00:15:35.690159 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.690039    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z" event={"ID":"4e2d4e66-888d-455b-bded-5db746148762","Type":"ContainerDied","Data":"9536692556bfaba6cb0d5ba03d940dc98589f328a3b7ea8ecd8317591603de30"}
Nov 03 00:15:35.690395 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.690381    5012 scope.go:122] "RemoveContainer" containerID="9536692556bfaba6cb0d5ba03d940dc98589f328a3b7ea8ecd8317591603de30"
Nov 03 00:15:35.695124 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.695109    5012 scope.go:122] "RemoveContainer" containerID="8b1a2a2237cf6ce5e431a0502ca8364c5adb98fd2bf150b21a72ffcb04622f59"
Nov 03 00:15:35.695363 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:15:35.695341    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b1a2a2237cf6ce5e431a0502ca8364c5adb98fd2bf150b21a72ffcb04622f59\": container with ID starting with 8b1a2a2237cf6ce5e431a0502ca8364c5adb98fd2bf150b21a72ffcb04622f59 not found: ID does not exist" containerID="8b1a2a2237cf6ce5e431a0502ca8364c5adb98fd2bf150b21a72ffcb04622f59"
Nov 03 00:15:35.695420 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.695375    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b1a2a2237cf6ce5e431a0502ca8364c5adb98fd2bf150b21a72ffcb04622f59"} err="failed to get container status \"8b1a2a2237cf6ce5e431a0502ca8364c5adb98fd2bf150b21a72ffcb04622f59\": rpc error: code = NotFound desc = could not find container \"8b1a2a2237cf6ce5e431a0502ca8364c5adb98fd2bf150b21a72ffcb04622f59\": container with ID starting with 8b1a2a2237cf6ce5e431a0502ca8364c5adb98fd2bf150b21a72ffcb04622f59 not found: ID does not exist"
Nov 03 00:15:35.695420 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.695396    5012 scope.go:122] "RemoveContainer" containerID="fc2a04537c5769df4ea375d2cbe8cc6b887d875968f98828355b760a6aba16f9"
Nov 03 00:15:35.695488 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.695419    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:15:35.695528 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.695512    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:15:35.695571 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.695557    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:15:35.708398 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.708382    5012 scope.go:122] "RemoveContainer" containerID="fc2a04537c5769df4ea375d2cbe8cc6b887d875968f98828355b760a6aba16f9"
Nov 03 00:15:35.708875 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:15:35.708842    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc2a04537c5769df4ea375d2cbe8cc6b887d875968f98828355b760a6aba16f9\": container with ID starting with fc2a04537c5769df4ea375d2cbe8cc6b887d875968f98828355b760a6aba16f9 not found: ID does not exist" containerID="fc2a04537c5769df4ea375d2cbe8cc6b887d875968f98828355b760a6aba16f9"
Nov 03 00:15:35.708975 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.708877    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc2a04537c5769df4ea375d2cbe8cc6b887d875968f98828355b760a6aba16f9"} err="failed to get container status \"fc2a04537c5769df4ea375d2cbe8cc6b887d875968f98828355b760a6aba16f9\": rpc error: code = NotFound desc = could not find container \"fc2a04537c5769df4ea375d2cbe8cc6b887d875968f98828355b760a6aba16f9\": container with ID starting with fc2a04537c5769df4ea375d2cbe8cc6b887d875968f98828355b760a6aba16f9 not found: ID does not exist"
Nov 03 00:15:35.734455 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.734433    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv"]
Nov 03 00:15:35.738105 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.738084    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c600f063-66b9c85fc5-p9rsv"]
Nov 03 00:15:35.769754 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.769734    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz"]
Nov 03 00:15:35.775286 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:35.775262    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c600f063-6dcbb79fc9-bhfjz"]
Nov 03 00:15:36.132039 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.132005    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54"]
Nov 03 00:15:36.132490 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.132474    5012 cpu_manager.go:414] "containerMap: removing container" podUID="ff82c8b4-dd1a-4870-829e-a37c98bf0280" containerName="payload-processing"
Nov 03 00:15:36.132575 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.132491    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff82c8b4-dd1a-4870-829e-a37c98bf0280" containerName="payload-processing"
Nov 03 00:15:36.132575 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.132508    5012 cpu_manager.go:414] "containerMap: removing container" podUID="aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e" containerName="cleanup"
Nov 03 00:15:36.132575 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.132516    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e" containerName="cleanup"
Nov 03 00:15:36.132575 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.132526    5012 cpu_manager.go:414] "containerMap: removing container" podUID="4d57748d-abae-487e-8e34-47ff716921be" containerName="payload-pre-processing"
Nov 03 00:15:36.132575 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.132555    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d57748d-abae-487e-8e34-47ff716921be" containerName="payload-pre-processing"
Nov 03 00:15:36.132575 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.132565    5012 cpu_manager.go:414] "containerMap: removing container" podUID="aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e" containerName="cleanup"
Nov 03 00:15:36.132575 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.132570    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e" containerName="cleanup"
Nov 03 00:15:36.132782 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.132582    5012 cpu_manager.go:414] "containerMap: removing container" podUID="dbf1c19b-8d46-401e-adb3-efa46f7dd7ff" containerName="payload-pre-processing"
Nov 03 00:15:36.132782 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.132587    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbf1c19b-8d46-401e-adb3-efa46f7dd7ff" containerName="payload-pre-processing"
Nov 03 00:15:36.132782 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.132598    5012 cpu_manager.go:414] "containerMap: removing container" podUID="a6bde7c6-1acf-4517-8698-2609d2c30872" containerName="payload-processing"
Nov 03 00:15:36.132782 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.132603    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6bde7c6-1acf-4517-8698-2609d2c30872" containerName="payload-processing"
Nov 03 00:15:36.132782 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.132613    5012 cpu_manager.go:414] "containerMap: removing container" podUID="aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e" containerName="cleanup"
Nov 03 00:15:36.132782 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.132621    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e" containerName="cleanup"
Nov 03 00:15:36.132782 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.132713    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e" containerName="cleanup"
Nov 03 00:15:36.132782 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.132722    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d57748d-abae-487e-8e34-47ff716921be" containerName="payload-pre-processing"
Nov 03 00:15:36.132782 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.132728    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff82c8b4-dd1a-4870-829e-a37c98bf0280" containerName="payload-processing"
Nov 03 00:15:36.132782 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.132733    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e" containerName="cleanup"
Nov 03 00:15:36.132782 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.132738    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbf1c19b-8d46-401e-adb3-efa46f7dd7ff" containerName="payload-pre-processing"
Nov 03 00:15:36.132782 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.132744    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6bde7c6-1acf-4517-8698-2609d2c30872" containerName="payload-processing"
Nov 03 00:15:36.136118 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.136099    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54"
Nov 03 00:15:36.150177 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.148846    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 00:15:36.150177 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.149003    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 00:15:36.151817 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.151792    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54"]
Nov 03 00:15:36.207935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.198173    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:15:36.207935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.198316    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:15:36.207935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.203016    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54\" (UID: \"9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54"
Nov 03 00:15:36.207935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.203069    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg7m5\" (UniqueName: \"kubernetes.io/projected/9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2-kube-api-access-lg7m5\") pod \"maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54\" (UID: \"9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54"
Nov 03 00:15:36.207935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.203170    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54\" (UID: \"9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54"
Nov 03 00:15:36.265227 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.265195    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"]
Nov 03 00:15:36.265441 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.265410    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z" podUID="c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" containerName="istio-proxy" containerID="cri-o://65d27ca51e212ea86cbf9b7e19f636692a4d0898f2d6398667e7850ecc96e586" gracePeriod=30
Nov 03 00:15:36.306934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.305497    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54\" (UID: \"9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54"
Nov 03 00:15:36.306934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.305639    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54\" (UID: \"9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54"
Nov 03 00:15:36.306934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.305670    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg7m5\" (UniqueName: \"kubernetes.io/projected/9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2-kube-api-access-lg7m5\") pod \"maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54\" (UID: \"9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54"
Nov 03 00:15:36.307239 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.307042    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54\" (UID: \"9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54"
Nov 03 00:15:36.313509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.313475    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54\" (UID: \"9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54"
Nov 03 00:15:36.325374 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.325351    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg7m5\" (UniqueName: \"kubernetes.io/projected/9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2-kube-api-access-lg7m5\") pod \"maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54\" (UID: \"9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54"
Nov 03 00:15:36.450305 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.450195    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54"
Nov 03 00:15:36.719803 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.719710    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z" event={"ID":"4e2d4e66-888d-455b-bded-5db746148762","Type":"ContainerStarted","Data":"e7e865a3993f59f97722109eaf3735e86e6664844789f2db0bcbf98a26dfea42"}
Nov 03 00:15:36.779830 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:36.779801    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54"]
Nov 03 00:15:36.780486 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:15:36.780463    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d84cb1b_5bfb_466a_b5c5_d3e9847dadf2.slice/crio-f2eecf71b029eb5abf402915846579aaeafe1c252db6de2581535c2bf1de813a WatchSource:0}: Error finding container f2eecf71b029eb5abf402915846579aaeafe1c252db6de2581535c2bf1de813a: Status 404 returned error can't find the container with id f2eecf71b029eb5abf402915846579aaeafe1c252db6de2581535c2bf1de813a
Nov 03 00:15:37.121352 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.121309    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"]
Nov 03 00:15:37.121627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.121599    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv" podUID="97953a4d-b832-4282-b010-d06cc1e7e3c2" containerName="istio-proxy" containerID="cri-o://63559afcdbc761b278af18d1a986f386e72ea2cbad2ff7005922d94fa42f80f3" gracePeriod=30
Nov 03 00:15:37.244206 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.244170    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d57748d-abae-487e-8e34-47ff716921be" path="/var/lib/kubelet/pods/4d57748d-abae-487e-8e34-47ff716921be/volumes"
Nov 03 00:15:37.244824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.244802    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff82c8b4-dd1a-4870-829e-a37c98bf0280" path="/var/lib/kubelet/pods/ff82c8b4-dd1a-4870-829e-a37c98bf0280/volumes"
Nov 03 00:15:37.349397 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.349366    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:37.350717 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.350692    5012 patch_prober.go:29] interesting pod/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 03 00:15:37.350853 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.350744    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z" podUID="4e2d4e66-888d-455b-bded-5db746148762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 03 00:15:37.364270 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:15:37.364242    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:15:37.461181 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.461151    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"]
Nov 03 00:15:37.462033 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.462008    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="aae16b30-ee5b-45f7-8d0e-ac5dda79bb8e" containerName="cleanup"
Nov 03 00:15:37.466860 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.466839    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.483252 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.483231    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-99bdb8-openshift-default-dockercfg-dq2f2\""
Nov 03 00:15:37.505939 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.502436    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"]
Nov 03 00:15:37.522718 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.522685    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.522848 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.522740    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.522848 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.522829    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.522994 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.522869    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.523032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.523015    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.523064 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.523034    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvdjh\" (UniqueName: \"kubernetes.io/projected/f474b6f6-e47d-4c13-8f63-b188ca20fce6-kube-api-access-mvdjh\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.523144 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.523072    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.523144 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.523125    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.523250 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.523154    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.624736 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.624647    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.624736 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.624685    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvdjh\" (UniqueName: \"kubernetes.io/projected/f474b6f6-e47d-4c13-8f63-b188ca20fce6-kube-api-access-mvdjh\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.624736 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.624720    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.624736 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.624737    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.625007 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.624753    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.625007 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.624791    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.625007 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.624805    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.625007 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.624825    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.625007 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.624841    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.625229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.625211    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.625425 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.625403    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.625795 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.625628    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.625795 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.625776    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.626062 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.626038    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.627831 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.627810    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.628605 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.628581    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.647450 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.647420    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvdjh\" (UniqueName: \"kubernetes.io/projected/f474b6f6-e47d-4c13-8f63-b188ca20fce6-kube-api-access-mvdjh\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.650742 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.650715    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:37.727230 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.727192    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54" event={"ID":"9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2","Type":"ContainerStarted","Data":"37576e9c0a41e354c31980b24bfb4dc8f9e7614740d965916a24ee8779999cf4"}
Nov 03 00:15:37.727690 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.727248    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54" event={"ID":"9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2","Type":"ContainerStarted","Data":"f2eecf71b029eb5abf402915846579aaeafe1c252db6de2581535c2bf1de813a"}
Nov 03 00:15:37.778384 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.778338    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54" podStartSLOduration=1.77832526 podStartE2EDuration="1.77832526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:37.772836243 +0000 UTC m=+620.652420287" watchObservedRunningTime="2026-11-03 00:15:37.77832526 +0000 UTC m=+620.657909303"
Nov 03 00:15:37.780105 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:37.780081    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:38.121344 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.121312    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"]
Nov 03 00:15:38.121767 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:15:38.121741    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf474b6f6_e47d_4c13_8f63_b188ca20fce6.slice/crio-5371106a9747a08fa18b3852a892b9effe08bd7b51913c638e6ddbc80bfee4df WatchSource:0}: Error finding container 5371106a9747a08fa18b3852a892b9effe08bd7b51913c638e6ddbc80bfee4df: Status 404 returned error can't find the container with id 5371106a9747a08fa18b3852a892b9effe08bd7b51913c638e6ddbc80bfee4df
Nov 03 00:15:38.124108 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.124077    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:15:38.124225 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.124154    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:15:38.124225 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.124201    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:15:38.341799 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.341765    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"]
Nov 03 00:15:38.345921 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.345887    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.349405 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.349025    5012 patch_prober.go:29] interesting pod/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 03 00:15:38.349405 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.349083    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z" podUID="4e2d4e66-888d-455b-bded-5db746148762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 03 00:15:38.363285 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.363266    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8e90c5-openshift-default-dockercfg-6mbt4\""
Nov 03 00:15:38.369649 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.369622    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"]
Nov 03 00:15:38.432297 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.432208    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.432463 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.432342    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t4gm\" (UniqueName: \"kubernetes.io/projected/4233c829-fa1c-443e-b7b6-9522c977fa98-kube-api-access-4t4gm\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.432463 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.432392    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.432463 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.432418    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.432463 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.432454    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4233c829-fa1c-443e-b7b6-9522c977fa98-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.432679 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.432496    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.432679 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.432530    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.432679 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.432566    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.432679 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.432647    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.533282 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.533244    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4233c829-fa1c-443e-b7b6-9522c977fa98-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.533749 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.533685    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.533981 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.533955    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.534142 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.534123    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.534289 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.534269    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4233c829-fa1c-443e-b7b6-9522c977fa98-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.534367 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.534021    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.534453 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.534186    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.536088 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.534577    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.538934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.536982    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.538934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.537072    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t4gm\" (UniqueName: \"kubernetes.io/projected/4233c829-fa1c-443e-b7b6-9522c977fa98-kube-api-access-4t4gm\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.538934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.537116    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.538934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.537141    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.538934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.538875    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.539408 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.539384    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.539504 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.539424    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.541565 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.541550    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.561449 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.561412    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.562255 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.562239    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t4gm\" (UniqueName: \"kubernetes.io/projected/4233c829-fa1c-443e-b7b6-9522c977fa98-kube-api-access-4t4gm\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.667542 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.667502    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:38.737699 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.737545    5012 generic.go:356] "Generic (PLEG): container finished" podID="9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2" containerID="37576e9c0a41e354c31980b24bfb4dc8f9e7614740d965916a24ee8779999cf4" exitCode=0
Nov 03 00:15:38.737699 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.737646    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54" event={"ID":"9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2","Type":"ContainerDied","Data":"37576e9c0a41e354c31980b24bfb4dc8f9e7614740d965916a24ee8779999cf4"}
Nov 03 00:15:38.740386 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.739973    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd" event={"ID":"f474b6f6-e47d-4c13-8f63-b188ca20fce6","Type":"ContainerStarted","Data":"72608dab623518b5be1416a046137a2d785eabfa85fd43ea68eda7942a2737c4"}
Nov 03 00:15:38.740386 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.740016    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd" event={"ID":"f474b6f6-e47d-4c13-8f63-b188ca20fce6","Type":"ContainerStarted","Data":"5371106a9747a08fa18b3852a892b9effe08bd7b51913c638e6ddbc80bfee4df"}
Nov 03 00:15:38.780852 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:38.780826    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:39.022978 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:39.016262    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd" podStartSLOduration=2.016240331 podStartE2EDuration="2.016240331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:38.832157817 +0000 UTC m=+621.711741862" watchObservedRunningTime="2026-11-03 00:15:39.016240331 +0000 UTC m=+621.895824377"
Nov 03 00:15:39.031475 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:39.031418    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"]
Nov 03 00:15:39.034541 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:39.034513    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:15:39.034626 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:39.034578    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:15:39.034626 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:39.034610    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:15:39.123158 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:39.123130    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v"]
Nov 03 00:15:39.156700 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:39.156665    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq"]
Nov 03 00:15:39.349413 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:39.349376    5012 patch_prober.go:29] interesting pod/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 03 00:15:39.349614 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:39.349447    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z" podUID="4e2d4e66-888d-455b-bded-5db746148762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 03 00:15:39.617975 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:39.617867    5012 patch_prober.go:29] interesting pod/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:15:39.617975 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:39.617947    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z" podUID="c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:15:39.754375 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:39.754340    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6" event={"ID":"4233c829-fa1c-443e-b7b6-9522c977fa98","Type":"ContainerStarted","Data":"b41b4b89c96bab9fe680c62048ca2a8bf0d67e17835e735f5ac578d87d9a287f"}
Nov 03 00:15:39.754878 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:39.754402    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6" event={"ID":"4233c829-fa1c-443e-b7b6-9522c977fa98","Type":"ContainerStarted","Data":"0d1b5507c2d019218bb65979167d26440a4fec74cc0cce932137284d2757c307"}
Nov 03 00:15:39.790937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:39.790212    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:39.856187 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:39.856140    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6" podStartSLOduration=1.8561276009999998 podStartE2EDuration="1.856127601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:39.847108248 +0000 UTC m=+622.726692316" watchObservedRunningTime="2026-11-03 00:15:39.856127601 +0000 UTC m=+622.735711644"
Nov 03 00:15:39.957961 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:39.957934    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54"
Nov 03 00:15:40.058933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.053616    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2-kube-api-access-lg7m5\" (UniqueName: \"kubernetes.io/projected/9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2-kube-api-access-lg7m5\") pod \"9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2\" (UID: \"9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2\") "
Nov 03 00:15:40.058933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.053668    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2-maas-api-openshift-ca\") pod \"9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2\" (UID: \"9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2\") "
Nov 03 00:15:40.058933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.053726    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2-maas-api-xks-ca\") pod \"9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2\" (UID: \"9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2\") "
Nov 03 00:15:40.058933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.054893    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2-maas-api-openshift-ca" pod "9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2" (UID: "9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:40.072747 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.070218    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2-kube-api-access-lg7m5" pod "9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2" (UID: "9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2"). InnerVolumeSpecName "kube-api-access-lg7m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:40.072747 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.070778    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2-maas-api-xks-ca" pod "9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2" (UID: "9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:15:40.155008 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.154974    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:40.155008 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.154999    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg7m5\" (UniqueName: \"kubernetes.io/projected/9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2-kube-api-access-lg7m5\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:40.155008 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.155009    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:40.267796 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.267714    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq" podUID="6443af0a-ce37-4412-b8e2-97d5a545ba52" containerName="payload-processing" containerID="cri-o://22e0c03014d4d9514fedb54be888b534b452b7030848ef533835d7ef5a50824b" gracePeriod=25
Nov 03 00:15:40.349036 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.349006    5012 patch_prober.go:29] interesting pod/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 03 00:15:40.349177 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.349056    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z" podUID="4e2d4e66-888d-455b-bded-5db746148762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 03 00:15:40.369423 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.369400    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:40.556345 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.556319    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq"
Nov 03 00:15:40.659738 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.659711    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6443af0a-ce37-4412-b8e2-97d5a545ba52-kube-api-access-whnfm\" (UniqueName: \"kubernetes.io/projected/6443af0a-ce37-4412-b8e2-97d5a545ba52-kube-api-access-whnfm\") pod \"6443af0a-ce37-4412-b8e2-97d5a545ba52\" (UID: \"6443af0a-ce37-4412-b8e2-97d5a545ba52\") "
Nov 03 00:15:40.660161 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.659841    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6443af0a-ce37-4412-b8e2-97d5a545ba52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6443af0a-ce37-4412-b8e2-97d5a545ba52-plugins-config-volume\") pod \"6443af0a-ce37-4412-b8e2-97d5a545ba52\" (UID: \"6443af0a-ce37-4412-b8e2-97d5a545ba52\") "
Nov 03 00:15:40.660231 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.660212    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6443af0a-ce37-4412-b8e2-97d5a545ba52-plugins-config-volume" pod "6443af0a-ce37-4412-b8e2-97d5a545ba52" (UID: "6443af0a-ce37-4412-b8e2-97d5a545ba52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:40.661779 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.661757    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6443af0a-ce37-4412-b8e2-97d5a545ba52-kube-api-access-whnfm" pod "6443af0a-ce37-4412-b8e2-97d5a545ba52" (UID: "6443af0a-ce37-4412-b8e2-97d5a545ba52"). InnerVolumeSpecName "kube-api-access-whnfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:40.667980 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.667963    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:40.672606 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.672587    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:40.760587 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.760560    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6443af0a-ce37-4412-b8e2-97d5a545ba52-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:40.760587 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.760586    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whnfm\" (UniqueName: \"kubernetes.io/projected/6443af0a-ce37-4412-b8e2-97d5a545ba52-kube-api-access-whnfm\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:40.761972 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.761947    5012 generic.go:356] "Generic (PLEG): container finished" podID="6443af0a-ce37-4412-b8e2-97d5a545ba52" containerID="22e0c03014d4d9514fedb54be888b534b452b7030848ef533835d7ef5a50824b" exitCode=0
Nov 03 00:15:40.762072 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.762059    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq"
Nov 03 00:15:40.762120 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.762084    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq" event={"ID":"6443af0a-ce37-4412-b8e2-97d5a545ba52","Type":"ContainerDied","Data":"22e0c03014d4d9514fedb54be888b534b452b7030848ef533835d7ef5a50824b"}
Nov 03 00:15:40.762174 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.762141    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq" event={"ID":"6443af0a-ce37-4412-b8e2-97d5a545ba52","Type":"ContainerDied","Data":"53df2c50e6328f19b9483a265959c7d9a42d586505faacfb78e0e415562fd983"}
Nov 03 00:15:40.762217 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.762172    5012 scope.go:122] "RemoveContainer" containerID="22e0c03014d4d9514fedb54be888b534b452b7030848ef533835d7ef5a50824b"
Nov 03 00:15:40.763575 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.763551    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54"
Nov 03 00:15:40.763702 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.763629    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54" event={"ID":"9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2","Type":"ContainerDied","Data":"f2eecf71b029eb5abf402915846579aaeafe1c252db6de2581535c2bf1de813a"}
Nov 03 00:15:40.763702 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.763667    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2eecf71b029eb5abf402915846579aaeafe1c252db6de2581535c2bf1de813a"
Nov 03 00:15:40.764285 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.764108    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:40.764285 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.764151    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:40.765353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.765335    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:15:40.765428 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.765407    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:15:40.773661 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.773630    5012 scope.go:122] "RemoveContainer" containerID="22e0c03014d4d9514fedb54be888b534b452b7030848ef533835d7ef5a50824b"
Nov 03 00:15:40.774067 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:15:40.774045    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22e0c03014d4d9514fedb54be888b534b452b7030848ef533835d7ef5a50824b\": container with ID starting with 22e0c03014d4d9514fedb54be888b534b452b7030848ef533835d7ef5a50824b not found: ID does not exist" containerID="22e0c03014d4d9514fedb54be888b534b452b7030848ef533835d7ef5a50824b"
Nov 03 00:15:40.774148 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.774078    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22e0c03014d4d9514fedb54be888b534b452b7030848ef533835d7ef5a50824b"} err="failed to get container status \"22e0c03014d4d9514fedb54be888b534b452b7030848ef533835d7ef5a50824b\": rpc error: code = NotFound desc = could not find container \"22e0c03014d4d9514fedb54be888b534b452b7030848ef533835d7ef5a50824b\": container with ID starting with 22e0c03014d4d9514fedb54be888b534b452b7030848ef533835d7ef5a50824b not found: ID does not exist"
Nov 03 00:15:40.914978 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.914944    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq"]
Nov 03 00:15:40.917942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:40.917892    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-19590f89-c669fd79c-7z6kq"]
Nov 03 00:15:41.068701 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.068141    5012 patch_prober.go:29] interesting pod/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:15:41.068701 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.068276    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv" podUID="97953a4d-b832-4282-b010-d06cc1e7e3c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:15:41.244095 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.244018    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6443af0a-ce37-4412-b8e2-97d5a545ba52" path="/var/lib/kubelet/pods/6443af0a-ce37-4412-b8e2-97d5a545ba52/volumes"
Nov 03 00:15:41.349573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.349543    5012 patch_prober.go:29] interesting pod/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 03 00:15:41.349715 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.349594    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z" podUID="4e2d4e66-888d-455b-bded-5db746148762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 03 00:15:41.600038 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.600010    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:15:41.670797 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.670762    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istiod-ca-cert\") pod \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") "
Nov 03 00:15:41.670989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.670880    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-kube-api-access-sjgxv\" (UniqueName: \"kubernetes.io/projected/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-kube-api-access-sjgxv\") pod \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") "
Nov 03 00:15:41.670989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.670954    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-credential-socket\") pod \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") "
Nov 03 00:15:41.671124 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.671005    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-workload-socket\") pod \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") "
Nov 03 00:15:41.671124 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.671041    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-podinfo\") pod \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") "
Nov 03 00:15:41.671124 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.671082    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-workload-certs\") pod \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") "
Nov 03 00:15:41.671124 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.671110    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-envoy\") pod \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") "
Nov 03 00:15:41.671326 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.671161    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-token\" (UniqueName: \"kubernetes.io/projected/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-token\") pod \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") "
Nov 03 00:15:41.671326 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.671199    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-data\") pod \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\" (UID: \"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8\") "
Nov 03 00:15:41.671326 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.671296    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-credential-socket" pod "c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" (UID: "c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:15:41.671479 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.671458    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-workload-certs" pod "c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" (UID: "c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:15:41.671538 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.671303    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-workload-socket" pod "c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" (UID: "c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:15:41.671590 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.671548    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-data" pod "c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" (UID: "c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:15:41.671676 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.671653    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:41.671676 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.671677    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:41.671857 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.671693    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:41.671857 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.671708    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:41.671989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.671972    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istiod-ca-cert" pod "c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" (UID: "c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:41.675646 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.674220    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-envoy" pod "c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" (UID: "c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:15:41.675646 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.674234    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-podinfo" pod "c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" (UID: "c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:15:41.675646 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.675197    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-token" pod "c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" (UID: "c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:41.675646 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.675432    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-kube-api-access-sjgxv" pod "c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" (UID: "c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8"). InnerVolumeSpecName "kube-api-access-sjgxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:41.772553 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.772514    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:41.772553 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.772552    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjgxv\" (UniqueName: \"kubernetes.io/projected/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-kube-api-access-sjgxv\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:41.773039 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.772569    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:41.773039 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.772583    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:41.773039 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.772597    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:41.775150 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.775116    5012 generic.go:356] "Generic (PLEG): container finished" podID="c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" containerID="65d27ca51e212ea86cbf9b7e19f636692a4d0898f2d6398667e7850ecc96e586" exitCode=0
Nov 03 00:15:41.775270 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.775184    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"
Nov 03 00:15:41.775270 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.775204    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z" event={"ID":"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8","Type":"ContainerDied","Data":"65d27ca51e212ea86cbf9b7e19f636692a4d0898f2d6398667e7850ecc96e586"}
Nov 03 00:15:41.775364 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.775314    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z" event={"ID":"c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8","Type":"ContainerDied","Data":"baa6c7de5ad0368dfb7f2c23aa03e5d8ea7dc5f7a2055e66ce877f939516c9d7"}
Nov 03 00:15:41.775364 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.775348    5012 scope.go:122] "RemoveContainer" containerID="65d27ca51e212ea86cbf9b7e19f636692a4d0898f2d6398667e7850ecc96e586"
Nov 03 00:15:41.788607 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.788589    5012 scope.go:122] "RemoveContainer" containerID="1cb8593e890e50a46b1b1df8ece05acf229d7e7740908ba472b6785a215be084"
Nov 03 00:15:41.799573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.799548    5012 scope.go:122] "RemoveContainer" containerID="65d27ca51e212ea86cbf9b7e19f636692a4d0898f2d6398667e7850ecc96e586"
Nov 03 00:15:41.799887 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:15:41.799861    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65d27ca51e212ea86cbf9b7e19f636692a4d0898f2d6398667e7850ecc96e586\": container with ID starting with 65d27ca51e212ea86cbf9b7e19f636692a4d0898f2d6398667e7850ecc96e586 not found: ID does not exist" containerID="65d27ca51e212ea86cbf9b7e19f636692a4d0898f2d6398667e7850ecc96e586"
Nov 03 00:15:41.799994 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.799900    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65d27ca51e212ea86cbf9b7e19f636692a4d0898f2d6398667e7850ecc96e586"} err="failed to get container status \"65d27ca51e212ea86cbf9b7e19f636692a4d0898f2d6398667e7850ecc96e586\": rpc error: code = NotFound desc = could not find container \"65d27ca51e212ea86cbf9b7e19f636692a4d0898f2d6398667e7850ecc96e586\": container with ID starting with 65d27ca51e212ea86cbf9b7e19f636692a4d0898f2d6398667e7850ecc96e586 not found: ID does not exist"
Nov 03 00:15:41.799994 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.799954    5012 scope.go:122] "RemoveContainer" containerID="1cb8593e890e50a46b1b1df8ece05acf229d7e7740908ba472b6785a215be084"
Nov 03 00:15:41.800255 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:15:41.800235    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cb8593e890e50a46b1b1df8ece05acf229d7e7740908ba472b6785a215be084\": container with ID starting with 1cb8593e890e50a46b1b1df8ece05acf229d7e7740908ba472b6785a215be084 not found: ID does not exist" containerID="1cb8593e890e50a46b1b1df8ece05acf229d7e7740908ba472b6785a215be084"
Nov 03 00:15:41.800255 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.800259    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cb8593e890e50a46b1b1df8ece05acf229d7e7740908ba472b6785a215be084"} err="failed to get container status \"1cb8593e890e50a46b1b1df8ece05acf229d7e7740908ba472b6785a215be084\": rpc error: code = NotFound desc = could not find container \"1cb8593e890e50a46b1b1df8ece05acf229d7e7740908ba472b6785a215be084\": container with ID starting with 1cb8593e890e50a46b1b1df8ece05acf229d7e7740908ba472b6785a215be084 not found: ID does not exist"
Nov 03 00:15:41.821490 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.821464    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"]
Nov 03 00:15:41.825633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:41.825581    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c600f063-openshift-default-5674c5ddb9-2zr4z"]
Nov 03 00:15:42.353652 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.353626    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:42.354919 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.354884    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:42.375366 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.375347    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:15:42.480831 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.480743    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-token\" (UniqueName: \"kubernetes.io/projected/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-token\") pod \"97953a4d-b832-4282-b010-d06cc1e7e3c2\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") "
Nov 03 00:15:42.480831 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.480823    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-workload-certs\") pod \"97953a4d-b832-4282-b010-d06cc1e7e3c2\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") "
Nov 03 00:15:42.481081 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.480893    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-workload-socket\") pod \"97953a4d-b832-4282-b010-d06cc1e7e3c2\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") "
Nov 03 00:15:42.481081 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.480957    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-envoy\") pod \"97953a4d-b832-4282-b010-d06cc1e7e3c2\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") "
Nov 03 00:15:42.481081 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.480987    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97953a4d-b832-4282-b010-d06cc1e7e3c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97953a4d-b832-4282-b010-d06cc1e7e3c2-istiod-ca-cert\") pod \"97953a4d-b832-4282-b010-d06cc1e7e3c2\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") "
Nov 03 00:15:42.481081 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.481029    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97953a4d-b832-4282-b010-d06cc1e7e3c2-kube-api-access-hk6z9\" (UniqueName: \"kubernetes.io/projected/97953a4d-b832-4282-b010-d06cc1e7e3c2-kube-api-access-hk6z9\") pod \"97953a4d-b832-4282-b010-d06cc1e7e3c2\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") "
Nov 03 00:15:42.481289 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.481082    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-data\") pod \"97953a4d-b832-4282-b010-d06cc1e7e3c2\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") "
Nov 03 00:15:42.481289 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.481132    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-podinfo\") pod \"97953a4d-b832-4282-b010-d06cc1e7e3c2\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") "
Nov 03 00:15:42.481289 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.481161    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-credential-socket\") pod \"97953a4d-b832-4282-b010-d06cc1e7e3c2\" (UID: \"97953a4d-b832-4282-b010-d06cc1e7e3c2\") "
Nov 03 00:15:42.481422 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.481349    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-workload-socket" pod "97953a4d-b832-4282-b010-d06cc1e7e3c2" (UID: "97953a4d-b832-4282-b010-d06cc1e7e3c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:15:42.482365 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.481575    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:42.482365 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.482187    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-workload-certs" pod "97953a4d-b832-4282-b010-d06cc1e7e3c2" (UID: "97953a4d-b832-4282-b010-d06cc1e7e3c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:15:42.482365 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.482304    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97953a4d-b832-4282-b010-d06cc1e7e3c2-istiod-ca-cert" pod "97953a4d-b832-4282-b010-d06cc1e7e3c2" (UID: "97953a4d-b832-4282-b010-d06cc1e7e3c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:42.482365 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.482152    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-data" pod "97953a4d-b832-4282-b010-d06cc1e7e3c2" (UID: "97953a4d-b832-4282-b010-d06cc1e7e3c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:15:42.482794 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.482734    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-credential-socket" pod "97953a4d-b832-4282-b010-d06cc1e7e3c2" (UID: "97953a4d-b832-4282-b010-d06cc1e7e3c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:15:42.484300 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.484271    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-token" pod "97953a4d-b832-4282-b010-d06cc1e7e3c2" (UID: "97953a4d-b832-4282-b010-d06cc1e7e3c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:42.486106 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.486082    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-envoy" pod "97953a4d-b832-4282-b010-d06cc1e7e3c2" (UID: "97953a4d-b832-4282-b010-d06cc1e7e3c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:15:42.486263 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.486241    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-podinfo" pod "97953a4d-b832-4282-b010-d06cc1e7e3c2" (UID: "97953a4d-b832-4282-b010-d06cc1e7e3c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:15:42.486351 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.486277    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97953a4d-b832-4282-b010-d06cc1e7e3c2-kube-api-access-hk6z9" pod "97953a4d-b832-4282-b010-d06cc1e7e3c2" (UID: "97953a4d-b832-4282-b010-d06cc1e7e3c2"). InnerVolumeSpecName "kube-api-access-hk6z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:42.589492 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.588010    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:42.589492 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.588043    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:42.589492 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.588056    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97953a4d-b832-4282-b010-d06cc1e7e3c2-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:42.589492 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.588070    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk6z9\" (UniqueName: \"kubernetes.io/projected/97953a4d-b832-4282-b010-d06cc1e7e3c2-kube-api-access-hk6z9\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:42.589492 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.588087    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:42.589492 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.588101    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:42.589492 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.588114    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97953a4d-b832-4282-b010-d06cc1e7e3c2-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:42.589492 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.588126    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97953a4d-b832-4282-b010-d06cc1e7e3c2-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:42.783381 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.783297    5012 generic.go:356] "Generic (PLEG): container finished" podID="97953a4d-b832-4282-b010-d06cc1e7e3c2" containerID="63559afcdbc761b278af18d1a986f386e72ea2cbad2ff7005922d94fa42f80f3" exitCode=0
Nov 03 00:15:42.783381 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.783353    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"
Nov 03 00:15:42.783880 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.783379    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv" event={"ID":"97953a4d-b832-4282-b010-d06cc1e7e3c2","Type":"ContainerDied","Data":"63559afcdbc761b278af18d1a986f386e72ea2cbad2ff7005922d94fa42f80f3"}
Nov 03 00:15:42.783880 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.783433    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv" event={"ID":"97953a4d-b832-4282-b010-d06cc1e7e3c2","Type":"ContainerDied","Data":"25c8bf3d0ad1ba0f0740ce70b597aefe20332e107376f1ae4a8c8de37b4c28dc"}
Nov 03 00:15:42.783880 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.783453    5012 scope.go:122] "RemoveContainer" containerID="63559afcdbc761b278af18d1a986f386e72ea2cbad2ff7005922d94fa42f80f3"
Nov 03 00:15:42.793397 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.793379    5012 scope.go:122] "RemoveContainer" containerID="6af6579a784d160b63a9ac72aabf7df7d1992c6e4a6a3505ce8607ed34d68cea"
Nov 03 00:15:42.802007 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.801985    5012 scope.go:122] "RemoveContainer" containerID="63559afcdbc761b278af18d1a986f386e72ea2cbad2ff7005922d94fa42f80f3"
Nov 03 00:15:42.802247 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:15:42.802229    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63559afcdbc761b278af18d1a986f386e72ea2cbad2ff7005922d94fa42f80f3\": container with ID starting with 63559afcdbc761b278af18d1a986f386e72ea2cbad2ff7005922d94fa42f80f3 not found: ID does not exist" containerID="63559afcdbc761b278af18d1a986f386e72ea2cbad2ff7005922d94fa42f80f3"
Nov 03 00:15:42.802307 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.802256    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63559afcdbc761b278af18d1a986f386e72ea2cbad2ff7005922d94fa42f80f3"} err="failed to get container status \"63559afcdbc761b278af18d1a986f386e72ea2cbad2ff7005922d94fa42f80f3\": rpc error: code = NotFound desc = could not find container \"63559afcdbc761b278af18d1a986f386e72ea2cbad2ff7005922d94fa42f80f3\": container with ID starting with 63559afcdbc761b278af18d1a986f386e72ea2cbad2ff7005922d94fa42f80f3 not found: ID does not exist"
Nov 03 00:15:42.802307 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.802271    5012 scope.go:122] "RemoveContainer" containerID="6af6579a784d160b63a9ac72aabf7df7d1992c6e4a6a3505ce8607ed34d68cea"
Nov 03 00:15:42.802540 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:15:42.802523    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6af6579a784d160b63a9ac72aabf7df7d1992c6e4a6a3505ce8607ed34d68cea\": container with ID starting with 6af6579a784d160b63a9ac72aabf7df7d1992c6e4a6a3505ce8607ed34d68cea not found: ID does not exist" containerID="6af6579a784d160b63a9ac72aabf7df7d1992c6e4a6a3505ce8607ed34d68cea"
Nov 03 00:15:42.802587 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.802544    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6af6579a784d160b63a9ac72aabf7df7d1992c6e4a6a3505ce8607ed34d68cea"} err="failed to get container status \"6af6579a784d160b63a9ac72aabf7df7d1992c6e4a6a3505ce8607ed34d68cea\": rpc error: code = NotFound desc = could not find container \"6af6579a784d160b63a9ac72aabf7df7d1992c6e4a6a3505ce8607ed34d68cea\": container with ID starting with 6af6579a784d160b63a9ac72aabf7df7d1992c6e4a6a3505ce8607ed34d68cea not found: ID does not exist"
Nov 03 00:15:42.827188 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.827168    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"]
Nov 03 00:15:42.837033 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:42.837013    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-b4cc5943-gw-openshift-default-6fd9d74777-7r8hv"]
Nov 03 00:15:43.156061 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.156026    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78"]
Nov 03 00:15:43.156462 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.156450    5012 cpu_manager.go:414] "containerMap: removing container" podUID="9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2" containerName="revoke-keys"
Nov 03 00:15:43.156506 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.156463    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2" containerName="revoke-keys"
Nov 03 00:15:43.156506 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.156476    5012 cpu_manager.go:414] "containerMap: removing container" podUID="c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" containerName="istio-proxy"
Nov 03 00:15:43.156506 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.156482    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" containerName="istio-proxy"
Nov 03 00:15:43.156506 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.156490    5012 cpu_manager.go:414] "containerMap: removing container" podUID="97953a4d-b832-4282-b010-d06cc1e7e3c2" containerName="istio-proxy"
Nov 03 00:15:43.156506 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.156494    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97953a4d-b832-4282-b010-d06cc1e7e3c2" containerName="istio-proxy"
Nov 03 00:15:43.156506 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.156504    5012 cpu_manager.go:414] "containerMap: removing container" podUID="6443af0a-ce37-4412-b8e2-97d5a545ba52" containerName="payload-processing"
Nov 03 00:15:43.156671 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.156508    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6443af0a-ce37-4412-b8e2-97d5a545ba52" containerName="payload-processing"
Nov 03 00:15:43.156671 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.156515    5012 cpu_manager.go:414] "containerMap: removing container" podUID="97953a4d-b832-4282-b010-d06cc1e7e3c2" containerName="istio-proxy"
Nov 03 00:15:43.156671 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.156519    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97953a4d-b832-4282-b010-d06cc1e7e3c2" containerName="istio-proxy"
Nov 03 00:15:43.156671 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.156543    5012 cpu_manager.go:414] "containerMap: removing container" podUID="c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" containerName="istio-proxy"
Nov 03 00:15:43.156671 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.156548    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" containerName="istio-proxy"
Nov 03 00:15:43.156671 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.156619    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" containerName="istio-proxy"
Nov 03 00:15:43.156671 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.156626    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="97953a4d-b832-4282-b010-d06cc1e7e3c2" containerName="istio-proxy"
Nov 03 00:15:43.156671 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.156633    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="6443af0a-ce37-4412-b8e2-97d5a545ba52" containerName="payload-processing"
Nov 03 00:15:43.156671 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.156640    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="97953a4d-b832-4282-b010-d06cc1e7e3c2" containerName="istio-proxy"
Nov 03 00:15:43.156671 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.156649    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2" containerName="revoke-keys"
Nov 03 00:15:43.161129 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.161115    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78"
Nov 03 00:15:43.176870 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.176844    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-99bdb8\""
Nov 03 00:15:43.177314 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.177295    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78"]
Nov 03 00:15:43.194525 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.194489    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36bbffa3-d72d-4f83-8916-d2d70377d2ea-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78\" (UID: \"36bbffa3-d72d-4f83-8916-d2d70377d2ea\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78"
Nov 03 00:15:43.194608 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.194534    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7p5k\" (UniqueName: \"kubernetes.io/projected/36bbffa3-d72d-4f83-8916-d2d70377d2ea-kube-api-access-b7p5k\") pod \"payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78\" (UID: \"36bbffa3-d72d-4f83-8916-d2d70377d2ea\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78"
Nov 03 00:15:43.227958 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.227935    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd"]
Nov 03 00:15:43.228437 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.228425    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" containerName="istio-proxy"
Nov 03 00:15:43.231625 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.231610    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd"
Nov 03 00:15:43.244087 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.244065    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97953a4d-b832-4282-b010-d06cc1e7e3c2" path="/var/lib/kubelet/pods/97953a4d-b832-4282-b010-d06cc1e7e3c2/volumes"
Nov 03 00:15:43.244510 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.244498    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8" path="/var/lib/kubelet/pods/c0d6a6d6-7eb1-40f1-83d3-547a3299fbc8/volumes"
Nov 03 00:15:43.248416 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.248397    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-99bdb8-dockercfg-chx9z\""
Nov 03 00:15:43.252153 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.252130    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd"]
Nov 03 00:15:43.295603 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.295573    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b420bc74-3aaf-4937-9237-9b35dc51ba8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd\" (UID: \"b420bc74-3aaf-4937-9237-9b35dc51ba8e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd"
Nov 03 00:15:43.295749 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.295637    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj9h9\" (UniqueName: \"kubernetes.io/projected/b420bc74-3aaf-4937-9237-9b35dc51ba8e-kube-api-access-nj9h9\") pod \"payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd\" (UID: \"b420bc74-3aaf-4937-9237-9b35dc51ba8e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd"
Nov 03 00:15:43.295749 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.295735    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36bbffa3-d72d-4f83-8916-d2d70377d2ea-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78\" (UID: \"36bbffa3-d72d-4f83-8916-d2d70377d2ea\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78"
Nov 03 00:15:43.295854 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.295769    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7p5k\" (UniqueName: \"kubernetes.io/projected/36bbffa3-d72d-4f83-8916-d2d70377d2ea-kube-api-access-b7p5k\") pod \"payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78\" (UID: \"36bbffa3-d72d-4f83-8916-d2d70377d2ea\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78"
Nov 03 00:15:43.296453 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.296430    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36bbffa3-d72d-4f83-8916-d2d70377d2ea-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78\" (UID: \"36bbffa3-d72d-4f83-8916-d2d70377d2ea\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78"
Nov 03 00:15:43.324849 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.324813    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7p5k\" (UniqueName: \"kubernetes.io/projected/36bbffa3-d72d-4f83-8916-d2d70377d2ea-kube-api-access-b7p5k\") pod \"payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78\" (UID: \"36bbffa3-d72d-4f83-8916-d2d70377d2ea\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78"
Nov 03 00:15:43.398153 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.398113    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj9h9\" (UniqueName: \"kubernetes.io/projected/b420bc74-3aaf-4937-9237-9b35dc51ba8e-kube-api-access-nj9h9\") pod \"payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd\" (UID: \"b420bc74-3aaf-4937-9237-9b35dc51ba8e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd"
Nov 03 00:15:43.398562 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.398534    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b420bc74-3aaf-4937-9237-9b35dc51ba8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd\" (UID: \"b420bc74-3aaf-4937-9237-9b35dc51ba8e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd"
Nov 03 00:15:43.399674 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.399636    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b420bc74-3aaf-4937-9237-9b35dc51ba8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd\" (UID: \"b420bc74-3aaf-4937-9237-9b35dc51ba8e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd"
Nov 03 00:15:43.428806 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.428728    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj9h9\" (UniqueName: \"kubernetes.io/projected/b420bc74-3aaf-4937-9237-9b35dc51ba8e-kube-api-access-nj9h9\") pod \"payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd\" (UID: \"b420bc74-3aaf-4937-9237-9b35dc51ba8e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd"
Nov 03 00:15:43.475761 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.475736    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78"
Nov 03 00:15:43.545505 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.545469    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd"
Nov 03 00:15:43.788533 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.788510    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78"]
Nov 03 00:15:43.790349 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:15:43.790320    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36bbffa3_d72d_4f83_8916_d2d70377d2ea.slice/crio-29877e9ab5566b0564404cd95d103ea7e7381bb09e152653922a8f980ee191d3 WatchSource:0}: Error finding container 29877e9ab5566b0564404cd95d103ea7e7381bb09e152653922a8f980ee191d3: Status 404 returned error can't find the container with id 29877e9ab5566b0564404cd95d103ea7e7381bb09e152653922a8f980ee191d3
Nov 03 00:15:43.848817 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:43.848788    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd"]
Nov 03 00:15:44.124147 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.124111    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v" podUID="f2da7b79-1d53-4f5b-a804-492511b2d75f" containerName="payload-processing" containerID="cri-o://455a0b7def04c809eb881d51a4f767e8fd433668cbb5fd14654189965eb9e324" gracePeriod=25
Nov 03 00:15:44.157550 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.157521    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq" podUID="b0147424-0774-4202-9291-da123886ff22" containerName="payload-pre-processing" containerID="cri-o://5265cb148b3035bca37b6f805ad5514a24f3aecd7c755ee8f361391a711d7ce6" gracePeriod=25
Nov 03 00:15:44.410648 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.410624    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v"
Nov 03 00:15:44.427007 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.426984    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq"
Nov 03 00:15:44.508355 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.508316    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0147424-0774-4202-9291-da123886ff22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0147424-0774-4202-9291-da123886ff22-plugins-config-volume\") pod \"b0147424-0774-4202-9291-da123886ff22\" (UID: \"b0147424-0774-4202-9291-da123886ff22\") "
Nov 03 00:15:44.508554 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.508403    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2da7b79-1d53-4f5b-a804-492511b2d75f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2da7b79-1d53-4f5b-a804-492511b2d75f-plugins-config-volume\") pod \"f2da7b79-1d53-4f5b-a804-492511b2d75f\" (UID: \"f2da7b79-1d53-4f5b-a804-492511b2d75f\") "
Nov 03 00:15:44.508554 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.508527    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2da7b79-1d53-4f5b-a804-492511b2d75f-kube-api-access-6vrxv\" (UniqueName: \"kubernetes.io/projected/f2da7b79-1d53-4f5b-a804-492511b2d75f-kube-api-access-6vrxv\") pod \"f2da7b79-1d53-4f5b-a804-492511b2d75f\" (UID: \"f2da7b79-1d53-4f5b-a804-492511b2d75f\") "
Nov 03 00:15:44.508682 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.508627    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0147424-0774-4202-9291-da123886ff22-kube-api-access-zvcqp\" (UniqueName: \"kubernetes.io/projected/b0147424-0774-4202-9291-da123886ff22-kube-api-access-zvcqp\") pod \"b0147424-0774-4202-9291-da123886ff22\" (UID: \"b0147424-0774-4202-9291-da123886ff22\") "
Nov 03 00:15:44.508825 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.508795    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0147424-0774-4202-9291-da123886ff22-plugins-config-volume" pod "b0147424-0774-4202-9291-da123886ff22" (UID: "b0147424-0774-4202-9291-da123886ff22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:44.509543 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.509514    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0147424-0774-4202-9291-da123886ff22-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:44.510737 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.509975    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2da7b79-1d53-4f5b-a804-492511b2d75f-plugins-config-volume" pod "f2da7b79-1d53-4f5b-a804-492511b2d75f" (UID: "f2da7b79-1d53-4f5b-a804-492511b2d75f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:44.511250 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.511212    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0147424-0774-4202-9291-da123886ff22-kube-api-access-zvcqp" pod "b0147424-0774-4202-9291-da123886ff22" (UID: "b0147424-0774-4202-9291-da123886ff22"). InnerVolumeSpecName "kube-api-access-zvcqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:44.511622 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.511596    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2da7b79-1d53-4f5b-a804-492511b2d75f-kube-api-access-6vrxv" pod "f2da7b79-1d53-4f5b-a804-492511b2d75f" (UID: "f2da7b79-1d53-4f5b-a804-492511b2d75f"). InnerVolumeSpecName "kube-api-access-6vrxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:44.610235 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.610204    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2da7b79-1d53-4f5b-a804-492511b2d75f-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:44.610235 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.610234    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vrxv\" (UniqueName: \"kubernetes.io/projected/f2da7b79-1d53-4f5b-a804-492511b2d75f-kube-api-access-6vrxv\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:44.610398 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.610247    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvcqp\" (UniqueName: \"kubernetes.io/projected/b0147424-0774-4202-9291-da123886ff22-kube-api-access-zvcqp\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:44.750393 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.750303    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx"]
Nov 03 00:15:44.750864 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.750846    5012 cpu_manager.go:414] "containerMap: removing container" podUID="b0147424-0774-4202-9291-da123886ff22" containerName="payload-pre-processing"
Nov 03 00:15:44.750985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.750866    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0147424-0774-4202-9291-da123886ff22" containerName="payload-pre-processing"
Nov 03 00:15:44.750985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.750883    5012 cpu_manager.go:414] "containerMap: removing container" podUID="f2da7b79-1d53-4f5b-a804-492511b2d75f" containerName="payload-processing"
Nov 03 00:15:44.750985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.750889    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2da7b79-1d53-4f5b-a804-492511b2d75f" containerName="payload-processing"
Nov 03 00:15:44.751152 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.751025    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0147424-0774-4202-9291-da123886ff22" containerName="payload-pre-processing"
Nov 03 00:15:44.751152 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.751040    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2da7b79-1d53-4f5b-a804-492511b2d75f" containerName="payload-processing"
Nov 03 00:15:44.754536 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.754515    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx"
Nov 03 00:15:44.767364 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.767346    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-8e90c5-dockercfg-7b62k\""
Nov 03 00:15:44.771145 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.771121    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx"]
Nov 03 00:15:44.801513 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.801475    5012 generic.go:356] "Generic (PLEG): container finished" podID="b0147424-0774-4202-9291-da123886ff22" containerID="5265cb148b3035bca37b6f805ad5514a24f3aecd7c755ee8f361391a711d7ce6" exitCode=0
Nov 03 00:15:44.801891 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.801512    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq"
Nov 03 00:15:44.801891 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.801537    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq" event={"ID":"b0147424-0774-4202-9291-da123886ff22","Type":"ContainerDied","Data":"5265cb148b3035bca37b6f805ad5514a24f3aecd7c755ee8f361391a711d7ce6"}
Nov 03 00:15:44.801891 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.801594    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq" event={"ID":"b0147424-0774-4202-9291-da123886ff22","Type":"ContainerDied","Data":"cc545e101719af1ca66440f286b640de017fa4c6de1b506c405ca36eae84a9d6"}
Nov 03 00:15:44.801891 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.801626    5012 scope.go:122] "RemoveContainer" containerID="5265cb148b3035bca37b6f805ad5514a24f3aecd7c755ee8f361391a711d7ce6"
Nov 03 00:15:44.804604 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.804584    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78" event={"ID":"36bbffa3-d72d-4f83-8916-d2d70377d2ea","Type":"ContainerStarted","Data":"eb84e8fe61df28aef04fe6c108b335306d0db57fe322d19cadbb146c0b7854ec"}
Nov 03 00:15:44.804694 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.804618    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78" event={"ID":"36bbffa3-d72d-4f83-8916-d2d70377d2ea","Type":"ContainerStarted","Data":"29877e9ab5566b0564404cd95d103ea7e7381bb09e152653922a8f980ee191d3"}
Nov 03 00:15:44.804694 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.804643    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78"
Nov 03 00:15:44.806137 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.806117    5012 generic.go:356] "Generic (PLEG): container finished" podID="f2da7b79-1d53-4f5b-a804-492511b2d75f" containerID="455a0b7def04c809eb881d51a4f767e8fd433668cbb5fd14654189965eb9e324" exitCode=0
Nov 03 00:15:44.806261 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.806151    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v"
Nov 03 00:15:44.806261 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.806202    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v" event={"ID":"f2da7b79-1d53-4f5b-a804-492511b2d75f","Type":"ContainerDied","Data":"455a0b7def04c809eb881d51a4f767e8fd433668cbb5fd14654189965eb9e324"}
Nov 03 00:15:44.806261 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.806247    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v" event={"ID":"f2da7b79-1d53-4f5b-a804-492511b2d75f","Type":"ContainerDied","Data":"28d9f2ed2cdd13c63e614021208800653b72224c294ed4d534441dbe7a5427c3"}
Nov 03 00:15:44.808075 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.808054    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd" event={"ID":"b420bc74-3aaf-4937-9237-9b35dc51ba8e","Type":"ContainerStarted","Data":"26a81f010246ba9b37556b94909b8c05c36f8c7a0d595de4ebbefb3008b7ca7c"}
Nov 03 00:15:44.808160 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.808094    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd" event={"ID":"b420bc74-3aaf-4937-9237-9b35dc51ba8e","Type":"ContainerStarted","Data":"2e5b5e39e49f997508e82ef637fb13ff49eb731181fc78c51a4bbad28d75dd23"}
Nov 03 00:15:44.808228 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.808192    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd"
Nov 03 00:15:44.812276 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.812253    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-8e90c5\""
Nov 03 00:15:44.812469 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.812447    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwxqn\" (UniqueName: \"kubernetes.io/projected/a314ceb2-941e-4c6d-8054-8c3bd7155692-kube-api-access-rwxqn\") pod \"payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx\" (UID: \"a314ceb2-941e-4c6d-8054-8c3bd7155692\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx"
Nov 03 00:15:44.812637 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.812604    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a314ceb2-941e-4c6d-8054-8c3bd7155692-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx\" (UID: \"a314ceb2-941e-4c6d-8054-8c3bd7155692\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx"
Nov 03 00:15:44.814059 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.814038    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8"]
Nov 03 00:15:44.814520 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.814494    5012 scope.go:122] "RemoveContainer" containerID="5265cb148b3035bca37b6f805ad5514a24f3aecd7c755ee8f361391a711d7ce6"
Nov 03 00:15:44.814786 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:15:44.814768    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5265cb148b3035bca37b6f805ad5514a24f3aecd7c755ee8f361391a711d7ce6\": container with ID starting with 5265cb148b3035bca37b6f805ad5514a24f3aecd7c755ee8f361391a711d7ce6 not found: ID does not exist" containerID="5265cb148b3035bca37b6f805ad5514a24f3aecd7c755ee8f361391a711d7ce6"
Nov 03 00:15:44.814870 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.814791    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5265cb148b3035bca37b6f805ad5514a24f3aecd7c755ee8f361391a711d7ce6"} err="failed to get container status \"5265cb148b3035bca37b6f805ad5514a24f3aecd7c755ee8f361391a711d7ce6\": rpc error: code = NotFound desc = could not find container \"5265cb148b3035bca37b6f805ad5514a24f3aecd7c755ee8f361391a711d7ce6\": container with ID starting with 5265cb148b3035bca37b6f805ad5514a24f3aecd7c755ee8f361391a711d7ce6 not found: ID does not exist"
Nov 03 00:15:44.814870 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.814807    5012 scope.go:122] "RemoveContainer" containerID="455a0b7def04c809eb881d51a4f767e8fd433668cbb5fd14654189965eb9e324"
Nov 03 00:15:44.819312 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.819292    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8"
Nov 03 00:15:44.824854 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.824840    5012 scope.go:122] "RemoveContainer" containerID="455a0b7def04c809eb881d51a4f767e8fd433668cbb5fd14654189965eb9e324"
Nov 03 00:15:44.825102 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:15:44.825082    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"455a0b7def04c809eb881d51a4f767e8fd433668cbb5fd14654189965eb9e324\": container with ID starting with 455a0b7def04c809eb881d51a4f767e8fd433668cbb5fd14654189965eb9e324 not found: ID does not exist" containerID="455a0b7def04c809eb881d51a4f767e8fd433668cbb5fd14654189965eb9e324"
Nov 03 00:15:44.825187 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.825107    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"455a0b7def04c809eb881d51a4f767e8fd433668cbb5fd14654189965eb9e324"} err="failed to get container status \"455a0b7def04c809eb881d51a4f767e8fd433668cbb5fd14654189965eb9e324\": rpc error: code = NotFound desc = could not find container \"455a0b7def04c809eb881d51a4f767e8fd433668cbb5fd14654189965eb9e324\": container with ID starting with 455a0b7def04c809eb881d51a4f767e8fd433668cbb5fd14654189965eb9e324 not found: ID does not exist"
Nov 03 00:15:44.834546 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.834525    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8"]
Nov 03 00:15:44.849703 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.849681    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v"]
Nov 03 00:15:44.853526 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.853505    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a96a129b-75cd9874fc-5896v"]
Nov 03 00:15:44.899081 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.899045    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq"]
Nov 03 00:15:44.900448 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.900419    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a96a129b-5c74588b59-fg7rq"]
Nov 03 00:15:44.913454 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.913419    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcb95377-3137-4676-b9b5-c464755a097f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8\" (UID: \"fcb95377-3137-4676-b9b5-c464755a097f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8"
Nov 03 00:15:44.913603 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.913527    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqt9z\" (UniqueName: \"kubernetes.io/projected/fcb95377-3137-4676-b9b5-c464755a097f-kube-api-access-sqt9z\") pod \"payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8\" (UID: \"fcb95377-3137-4676-b9b5-c464755a097f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8"
Nov 03 00:15:44.913680 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.913637    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a314ceb2-941e-4c6d-8054-8c3bd7155692-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx\" (UID: \"a314ceb2-941e-4c6d-8054-8c3bd7155692\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx"
Nov 03 00:15:44.913734 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.913697    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwxqn\" (UniqueName: \"kubernetes.io/projected/a314ceb2-941e-4c6d-8054-8c3bd7155692-kube-api-access-rwxqn\") pod \"payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx\" (UID: \"a314ceb2-941e-4c6d-8054-8c3bd7155692\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx"
Nov 03 00:15:44.914451 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.914428    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a314ceb2-941e-4c6d-8054-8c3bd7155692-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx\" (UID: \"a314ceb2-941e-4c6d-8054-8c3bd7155692\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx"
Nov 03 00:15:44.922429 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.922383    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78" podStartSLOduration=1.922372019 podStartE2EDuration="1.922372019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:44.921601454 +0000 UTC m=+627.801185499" watchObservedRunningTime="2026-11-03 00:15:44.922372019 +0000 UTC m=+627.801956059"
Nov 03 00:15:44.932604 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.932583    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwxqn\" (UniqueName: \"kubernetes.io/projected/a314ceb2-941e-4c6d-8054-8c3bd7155692-kube-api-access-rwxqn\") pod \"payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx\" (UID: \"a314ceb2-941e-4c6d-8054-8c3bd7155692\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx"
Nov 03 00:15:44.955077 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:44.955033    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd" podStartSLOduration=1.955022488 podStartE2EDuration="1.955022488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:44.953748638 +0000 UTC m=+627.833332681" watchObservedRunningTime="2026-11-03 00:15:44.955022488 +0000 UTC m=+627.834606531"
Nov 03 00:15:45.015031 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:45.014972    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcb95377-3137-4676-b9b5-c464755a097f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8\" (UID: \"fcb95377-3137-4676-b9b5-c464755a097f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8"
Nov 03 00:15:45.015132 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:45.015068    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqt9z\" (UniqueName: \"kubernetes.io/projected/fcb95377-3137-4676-b9b5-c464755a097f-kube-api-access-sqt9z\") pod \"payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8\" (UID: \"fcb95377-3137-4676-b9b5-c464755a097f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8"
Nov 03 00:15:45.015667 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:45.015647    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcb95377-3137-4676-b9b5-c464755a097f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8\" (UID: \"fcb95377-3137-4676-b9b5-c464755a097f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8"
Nov 03 00:15:45.031965 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:45.031943    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqt9z\" (UniqueName: \"kubernetes.io/projected/fcb95377-3137-4676-b9b5-c464755a097f-kube-api-access-sqt9z\") pod \"payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8\" (UID: \"fcb95377-3137-4676-b9b5-c464755a097f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8"
Nov 03 00:15:45.067527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:45.067294    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx"
Nov 03 00:15:45.135230 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:45.135196    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8"
Nov 03 00:15:45.246088 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:45.246057    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0147424-0774-4202-9291-da123886ff22" path="/var/lib/kubelet/pods/b0147424-0774-4202-9291-da123886ff22/volumes"
Nov 03 00:15:45.246456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:45.246442    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2da7b79-1d53-4f5b-a804-492511b2d75f" path="/var/lib/kubelet/pods/f2da7b79-1d53-4f5b-a804-492511b2d75f/volumes"
Nov 03 00:15:45.369435 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:45.369407    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx"]
Nov 03 00:15:45.370598 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:15:45.370567    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda314ceb2_941e_4c6d_8054_8c3bd7155692.slice/crio-b04d6c7f2e81a5402b2795c696ec6b513353c2df040c9cb30b3cd1872faa83bc WatchSource:0}: Error finding container b04d6c7f2e81a5402b2795c696ec6b513353c2df040c9cb30b3cd1872faa83bc: Status 404 returned error can't find the container with id b04d6c7f2e81a5402b2795c696ec6b513353c2df040c9cb30b3cd1872faa83bc
Nov 03 00:15:45.442946 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:45.442904    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8"]
Nov 03 00:15:45.446794 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:15:45.446762    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb95377_3137_4676_b9b5_c464755a097f.slice/crio-10d35c58a161230f12503919dfa7f5be9fb3c5151f0ee3b334c6498929389a22 WatchSource:0}: Error finding container 10d35c58a161230f12503919dfa7f5be9fb3c5151f0ee3b334c6498929389a22: Status 404 returned error can't find the container with id 10d35c58a161230f12503919dfa7f5be9fb3c5151f0ee3b334c6498929389a22
Nov 03 00:15:45.824127 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:45.823237    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx" event={"ID":"a314ceb2-941e-4c6d-8054-8c3bd7155692","Type":"ContainerStarted","Data":"c60e8b81a090637a7da6b66c057b52c2e5854c4d522e6d1fb94fb47ae14bf7ed"}
Nov 03 00:15:45.824127 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:45.823313    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx" event={"ID":"a314ceb2-941e-4c6d-8054-8c3bd7155692","Type":"ContainerStarted","Data":"b04d6c7f2e81a5402b2795c696ec6b513353c2df040c9cb30b3cd1872faa83bc"}
Nov 03 00:15:45.824127 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:45.823503    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx"
Nov 03 00:15:45.835577 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:45.835535    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8" event={"ID":"fcb95377-3137-4676-b9b5-c464755a097f","Type":"ContainerStarted","Data":"487d2b26fca7ecc4f5d8bb3e90f30c7dfaa79995c9b983ece599f6747ba904f3"}
Nov 03 00:15:45.835755 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:45.835592    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8" event={"ID":"fcb95377-3137-4676-b9b5-c464755a097f","Type":"ContainerStarted","Data":"10d35c58a161230f12503919dfa7f5be9fb3c5151f0ee3b334c6498929389a22"}
Nov 03 00:15:45.835755 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:45.835619    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8"
Nov 03 00:15:45.864082 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:45.864034    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx" podStartSLOduration=1.864020466 podStartE2EDuration="1.864020466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:45.86111185 +0000 UTC m=+628.740695895" watchObservedRunningTime="2026-11-03 00:15:45.864020466 +0000 UTC m=+628.743604505"
Nov 03 00:15:45.898876 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:45.898829    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8" podStartSLOduration=1.898815794 podStartE2EDuration="1.898815794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:45.897261245 +0000 UTC m=+628.776845289" watchObservedRunningTime="2026-11-03 00:15:45.898815794 +0000 UTC m=+628.778399836"
Nov 03 00:15:46.476452 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:46.476090    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2"]
Nov 03 00:15:47.882036 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:47.881995    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw"]
Nov 03 00:15:47.886409 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:47.886388    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb"]
Nov 03 00:15:47.886544 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:47.886500    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw"
Nov 03 00:15:47.889803 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:47.889784    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb"
Nov 03 00:15:47.902763 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:47.902745    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-fc9316\""
Nov 03 00:15:47.902969 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:47.902947    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw"]
Nov 03 00:15:47.912586 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:47.912564    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb"]
Nov 03 00:15:47.958692 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:47.958668    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw\" (UID: \"bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw"
Nov 03 00:15:47.958832 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:47.958696    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lstlx\" (UniqueName: \"kubernetes.io/projected/bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4-kube-api-access-lstlx\") pod \"payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw\" (UID: \"bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw"
Nov 03 00:15:47.958832 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:47.958722    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blns4\" (UniqueName: \"kubernetes.io/projected/373c74f5-bc8f-4938-b64f-f4ab8d6bb993-kube-api-access-blns4\") pod \"payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb\" (UID: \"373c74f5-bc8f-4938-b64f-f4ab8d6bb993\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb"
Nov 03 00:15:47.959041 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:47.958859    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373c74f5-bc8f-4938-b64f-f4ab8d6bb993-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb\" (UID: \"373c74f5-bc8f-4938-b64f-f4ab8d6bb993\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb"
Nov 03 00:15:48.060049 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:48.060006    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw\" (UID: \"bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw"
Nov 03 00:15:48.060229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:48.060155    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lstlx\" (UniqueName: \"kubernetes.io/projected/bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4-kube-api-access-lstlx\") pod \"payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw\" (UID: \"bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw"
Nov 03 00:15:48.060279 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:48.060223    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blns4\" (UniqueName: \"kubernetes.io/projected/373c74f5-bc8f-4938-b64f-f4ab8d6bb993-kube-api-access-blns4\") pod \"payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb\" (UID: \"373c74f5-bc8f-4938-b64f-f4ab8d6bb993\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb"
Nov 03 00:15:48.060318 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:48.060299    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373c74f5-bc8f-4938-b64f-f4ab8d6bb993-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb\" (UID: \"373c74f5-bc8f-4938-b64f-f4ab8d6bb993\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb"
Nov 03 00:15:48.060822 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:48.060802    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw\" (UID: \"bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw"
Nov 03 00:15:48.061133 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:48.061114    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373c74f5-bc8f-4938-b64f-f4ab8d6bb993-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb\" (UID: \"373c74f5-bc8f-4938-b64f-f4ab8d6bb993\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb"
Nov 03 00:15:48.078723 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:48.078692    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blns4\" (UniqueName: \"kubernetes.io/projected/373c74f5-bc8f-4938-b64f-f4ab8d6bb993-kube-api-access-blns4\") pod \"payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb\" (UID: \"373c74f5-bc8f-4938-b64f-f4ab8d6bb993\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb"
Nov 03 00:15:48.078723 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:48.078713    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lstlx\" (UniqueName: \"kubernetes.io/projected/bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4-kube-api-access-lstlx\") pod \"payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw\" (UID: \"bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw"
Nov 03 00:15:48.203848 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:48.203791    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw"
Nov 03 00:15:48.209823 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:48.209802    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb"
Nov 03 00:15:48.524905 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:48.524870    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw"]
Nov 03 00:15:48.529309 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:48.529256    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb"]
Nov 03 00:15:48.532469 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:15:48.531516    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373c74f5_bc8f_4938_b64f_f4ab8d6bb993.slice/crio-070fcf4f14f738f8802ff25f6ae0e5f63ecd88709168b99dd7be351b53f8c325 WatchSource:0}: Error finding container 070fcf4f14f738f8802ff25f6ae0e5f63ecd88709168b99dd7be351b53f8c325: Status 404 returned error can't find the container with id 070fcf4f14f738f8802ff25f6ae0e5f63ecd88709168b99dd7be351b53f8c325
Nov 03 00:15:48.871747 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:48.871698    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw" event={"ID":"bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4","Type":"ContainerStarted","Data":"627544b5c7d32d09faf4b80b0fd860d862727c1461723655ee351598b2e01f01"}
Nov 03 00:15:48.871947 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:48.871792    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw" event={"ID":"bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4","Type":"ContainerStarted","Data":"7c065384b4612d9b866e7632b092a7646a4dd2c53040ecbbfcf82d302106f4f4"}
Nov 03 00:15:48.871947 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:48.871824    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw"
Nov 03 00:15:48.873387 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:48.873354    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb" event={"ID":"373c74f5-bc8f-4938-b64f-f4ab8d6bb993","Type":"ContainerStarted","Data":"71a8b09dd01a0eba38aecd97932469c88015e2b204465eaae74b791e78085709"}
Nov 03 00:15:48.873502 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:48.873411    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb"
Nov 03 00:15:48.873502 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:48.873435    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb" event={"ID":"373c74f5-bc8f-4938-b64f-f4ab8d6bb993","Type":"ContainerStarted","Data":"070fcf4f14f738f8802ff25f6ae0e5f63ecd88709168b99dd7be351b53f8c325"}
Nov 03 00:15:48.909068 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:48.909021    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw" podStartSLOduration=1.9090067149999999 podStartE2EDuration="1.909006715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:48.908813042 +0000 UTC m=+631.788397084" watchObservedRunningTime="2026-11-03 00:15:48.909006715 +0000 UTC m=+631.788590756"
Nov 03 00:15:48.946724 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:48.946680    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb" podStartSLOduration=1.946668292 podStartE2EDuration="1.946668292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:48.945247956 +0000 UTC m=+631.824831998" watchObservedRunningTime="2026-11-03 00:15:48.946668292 +0000 UTC m=+631.826252335"
Nov 03 00:15:50.188923 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:50.188871    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957"]
Nov 03 00:15:50.193863 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:50.193834    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957"
Nov 03 00:15:50.244931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:50.244210    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957"]
Nov 03 00:15:50.282071 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:50.282037    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-4c61c8\""
Nov 03 00:15:50.282430 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:50.282408    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-4c61c8-dockercfg-dmk6d\""
Nov 03 00:15:50.298928 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:50.284295    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bshwm\" (UniqueName: \"kubernetes.io/projected/a325b3a8-8c15-40f2-8f63-47e0e15795a6-kube-api-access-bshwm\") pod \"payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957\" (UID: \"a325b3a8-8c15-40f2-8f63-47e0e15795a6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957"
Nov 03 00:15:50.299237 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:50.299204    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a325b3a8-8c15-40f2-8f63-47e0e15795a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957\" (UID: \"a325b3a8-8c15-40f2-8f63-47e0e15795a6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957"
Nov 03 00:15:50.401004 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:50.400969    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bshwm\" (UniqueName: \"kubernetes.io/projected/a325b3a8-8c15-40f2-8f63-47e0e15795a6-kube-api-access-bshwm\") pod \"payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957\" (UID: \"a325b3a8-8c15-40f2-8f63-47e0e15795a6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957"
Nov 03 00:15:50.401180 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:50.401063    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a325b3a8-8c15-40f2-8f63-47e0e15795a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957\" (UID: \"a325b3a8-8c15-40f2-8f63-47e0e15795a6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957"
Nov 03 00:15:50.401945 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:50.401894    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a325b3a8-8c15-40f2-8f63-47e0e15795a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957\" (UID: \"a325b3a8-8c15-40f2-8f63-47e0e15795a6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957"
Nov 03 00:15:50.418289 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:50.418269    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bshwm\" (UniqueName: \"kubernetes.io/projected/a325b3a8-8c15-40f2-8f63-47e0e15795a6-kube-api-access-bshwm\") pod \"payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957\" (UID: \"a325b3a8-8c15-40f2-8f63-47e0e15795a6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957"
Nov 03 00:15:50.508511 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:50.508430    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957"
Nov 03 00:15:50.844213 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:50.844184    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957"]
Nov 03 00:15:50.845535 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:15:50.845513    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda325b3a8_8c15_40f2_8f63_47e0e15795a6.slice/crio-157c4a309d910a142fd1c0238d42507007ffe11d7b89167d14723c6c74398c2d WatchSource:0}: Error finding container 157c4a309d910a142fd1c0238d42507007ffe11d7b89167d14723c6c74398c2d: Status 404 returned error can't find the container with id 157c4a309d910a142fd1c0238d42507007ffe11d7b89167d14723c6c74398c2d
Nov 03 00:15:50.886652 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:50.886623    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957" event={"ID":"a325b3a8-8c15-40f2-8f63-47e0e15795a6","Type":"ContainerStarted","Data":"157c4a309d910a142fd1c0238d42507007ffe11d7b89167d14723c6c74398c2d"}
Nov 03 00:15:51.477389 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:51.477339    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2" podUID="a168116a-4202-4d6d-9007-e54f691c1a76" containerName="payload-processing" containerID="cri-o://6c441df31afb23373b4b8ab4a0042680c912c7a2bf99b1576509077f21d2893a" gracePeriod=25
Nov 03 00:15:51.883976 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:51.883949    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2"
Nov 03 00:15:51.937933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:51.936983    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a168116a-4202-4d6d-9007-e54f691c1a76-kube-api-access-r2wq6\" (UniqueName: \"kubernetes.io/projected/a168116a-4202-4d6d-9007-e54f691c1a76-kube-api-access-r2wq6\") pod \"a168116a-4202-4d6d-9007-e54f691c1a76\" (UID: \"a168116a-4202-4d6d-9007-e54f691c1a76\") "
Nov 03 00:15:51.942059 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:51.939653    5012 generic.go:356] "Generic (PLEG): container finished" podID="a168116a-4202-4d6d-9007-e54f691c1a76" containerID="6c441df31afb23373b4b8ab4a0042680c912c7a2bf99b1576509077f21d2893a" exitCode=0
Nov 03 00:15:51.942059 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:51.939754    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2"
Nov 03 00:15:51.942059 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:51.939774    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2" event={"ID":"a168116a-4202-4d6d-9007-e54f691c1a76","Type":"ContainerDied","Data":"6c441df31afb23373b4b8ab4a0042680c912c7a2bf99b1576509077f21d2893a"}
Nov 03 00:15:51.942059 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:51.940035    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2" event={"ID":"a168116a-4202-4d6d-9007-e54f691c1a76","Type":"ContainerDied","Data":"cdecfc09ccd7cba7c828d28fa0d478bd47259c8b1eedb2edef12347fdb206076"}
Nov 03 00:15:51.942059 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:51.940066    5012 scope.go:122] "RemoveContainer" containerID="6c441df31afb23373b4b8ab4a0042680c912c7a2bf99b1576509077f21d2893a"
Nov 03 00:15:51.942059 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:51.941028    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a168116a-4202-4d6d-9007-e54f691c1a76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a168116a-4202-4d6d-9007-e54f691c1a76-plugins-config-volume\") pod \"a168116a-4202-4d6d-9007-e54f691c1a76\" (UID: \"a168116a-4202-4d6d-9007-e54f691c1a76\") "
Nov 03 00:15:51.942059 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:51.941887    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a168116a-4202-4d6d-9007-e54f691c1a76-plugins-config-volume" pod "a168116a-4202-4d6d-9007-e54f691c1a76" (UID: "a168116a-4202-4d6d-9007-e54f691c1a76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:51.948628 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:51.948570    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957" event={"ID":"a325b3a8-8c15-40f2-8f63-47e0e15795a6","Type":"ContainerStarted","Data":"7552ec53e6e63a469bcce528b9153fff6ea1f77065c47d8711b22c6721a5464a"}
Nov 03 00:15:51.948816 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:51.948780    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957"
Nov 03 00:15:51.949338 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:51.949313    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a168116a-4202-4d6d-9007-e54f691c1a76-kube-api-access-r2wq6" pod "a168116a-4202-4d6d-9007-e54f691c1a76" (UID: "a168116a-4202-4d6d-9007-e54f691c1a76"). InnerVolumeSpecName "kube-api-access-r2wq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:51.956453 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:51.956213    5012 scope.go:122] "RemoveContainer" containerID="6c441df31afb23373b4b8ab4a0042680c912c7a2bf99b1576509077f21d2893a"
Nov 03 00:15:51.956554 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:15:51.956501    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c441df31afb23373b4b8ab4a0042680c912c7a2bf99b1576509077f21d2893a\": container with ID starting with 6c441df31afb23373b4b8ab4a0042680c912c7a2bf99b1576509077f21d2893a not found: ID does not exist" containerID="6c441df31afb23373b4b8ab4a0042680c912c7a2bf99b1576509077f21d2893a"
Nov 03 00:15:51.956605 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:51.956549    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c441df31afb23373b4b8ab4a0042680c912c7a2bf99b1576509077f21d2893a"} err="failed to get container status \"6c441df31afb23373b4b8ab4a0042680c912c7a2bf99b1576509077f21d2893a\": rpc error: code = NotFound desc = could not find container \"6c441df31afb23373b4b8ab4a0042680c912c7a2bf99b1576509077f21d2893a\": container with ID starting with 6c441df31afb23373b4b8ab4a0042680c912c7a2bf99b1576509077f21d2893a not found: ID does not exist"
Nov 03 00:15:52.042615 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:52.042513    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2wq6\" (UniqueName: \"kubernetes.io/projected/a168116a-4202-4d6d-9007-e54f691c1a76-kube-api-access-r2wq6\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:52.042615 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:52.042543    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a168116a-4202-4d6d-9007-e54f691c1a76-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:52.278700 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:52.278651    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957" podStartSLOduration=2.278635704 podStartE2EDuration="2.278635704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:52.009213333 +0000 UTC m=+634.888797403" watchObservedRunningTime="2026-11-03 00:15:52.278635704 +0000 UTC m=+635.158219746"
Nov 03 00:15:52.281074 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:52.281052    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2"]
Nov 03 00:15:52.302987 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:52.299979    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e3e6ed37-668988b567-snsh2"]
Nov 03 00:15:53.254397 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:53.254365    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a168116a-4202-4d6d-9007-e54f691c1a76" path="/var/lib/kubelet/pods/a168116a-4202-4d6d-9007-e54f691c1a76/volumes"
Nov 03 00:15:54.035944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:54.032773    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54"]
Nov 03 00:15:54.040938 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:54.040471    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e3e6ed37-tng54"]
Nov 03 00:15:54.157862 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:54.157827    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"]
Nov 03 00:15:54.158183 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:54.158148    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z" podUID="4e2d4e66-888d-455b-bded-5db746148762" containerName="istio-proxy" containerID="cri-o://e7e865a3993f59f97722109eaf3735e86e6664844789f2db0bcbf98a26dfea42" gracePeriod=30
Nov 03 00:15:55.248805 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.248768    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2" path="/var/lib/kubelet/pods/9d84cb1b-5bfb-466a-b5c5-d3e9847dadf2/volumes"
Nov 03 00:15:55.457013 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.456979    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"]
Nov 03 00:15:55.457997 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.457880    5012 cpu_manager.go:414] "containerMap: removing container" podUID="a168116a-4202-4d6d-9007-e54f691c1a76" containerName="payload-processing"
Nov 03 00:15:55.459139 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.459116    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a168116a-4202-4d6d-9007-e54f691c1a76" containerName="payload-processing"
Nov 03 00:15:55.459391 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.459377    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="a168116a-4202-4d6d-9007-e54f691c1a76" containerName="payload-processing"
Nov 03 00:15:55.464887 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.464867    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"]
Nov 03 00:15:55.465002 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.464977    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.477869 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.477845    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b85343-openshift-default-dockercfg-jnjww\""
Nov 03 00:15:55.479471 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.479297    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a090865-de87-442c-806a-76e91d5d374c-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.479471 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.479340    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.479471 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.479378    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a090865-de87-442c-806a-76e91d5d374c-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.479471 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.479415    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a090865-de87-442c-806a-76e91d5d374c-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.479471 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.479462    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.479814 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.479513    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.479814 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.479543    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.479814 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.479578    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.479814 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.479639    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v252f\" (UniqueName: \"kubernetes.io/projected/8a090865-de87-442c-806a-76e91d5d374c-kube-api-access-v252f\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.581197 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.581112    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a090865-de87-442c-806a-76e91d5d374c-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.581197 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.581155    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.581197 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.581194    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a090865-de87-442c-806a-76e91d5d374c-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.581469 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.581228    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a090865-de87-442c-806a-76e91d5d374c-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.581469 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.581345    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.581469 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.581402    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.581636 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.581559    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.581902 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.581861    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.582057 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.581935    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.582057 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.582000    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v252f\" (UniqueName: \"kubernetes.io/projected/8a090865-de87-442c-806a-76e91d5d374c-kube-api-access-v252f\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.582190 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.582052    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.582190 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.582052    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a090865-de87-442c-806a-76e91d5d374c-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.582313 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.582268    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.582434 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.582410    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.583761 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.583742    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a090865-de87-442c-806a-76e91d5d374c-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.585366 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.585340    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.619564 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.619493    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a090865-de87-442c-806a-76e91d5d374c-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.624932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.621185    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v252f\" (UniqueName: \"kubernetes.io/projected/8a090865-de87-442c-806a-76e91d5d374c-kube-api-access-v252f\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.781604 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.781389    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:55.846997 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.846723    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd"
Nov 03 00:15:55.846997 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:55.846823    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78"
Nov 03 00:15:56.239922 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:56.239858    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"]
Nov 03 00:15:56.246143 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:56.246109    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:15:56.246224 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:56.246186    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:15:56.246258 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:56.246216    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:15:56.860056 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:56.860028    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx"
Nov 03 00:15:56.860436 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:56.860098    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8"
Nov 03 00:15:57.030493 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:57.030448    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9" event={"ID":"8a090865-de87-442c-806a-76e91d5d374c","Type":"ContainerStarted","Data":"c910e4fab82b248bac75a840eca104a7ab622e2d9564d41f36ffbc91848e546a"}
Nov 03 00:15:57.030664 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:57.030519    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9" event={"ID":"8a090865-de87-442c-806a-76e91d5d374c","Type":"ContainerStarted","Data":"4a92e65840bf5c815e336c6be7ba4ca576fd047721c899dadf15f6c5c04895c6"}
Nov 03 00:15:57.077114 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:57.077061    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9" podStartSLOduration=2.07704576 podStartE2EDuration="2.07704576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:57.07621295 +0000 UTC m=+639.955796994" watchObservedRunningTime="2026-11-03 00:15:57.07704576 +0000 UTC m=+639.956629803"
Nov 03 00:15:57.356733 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:57.356693    5012 patch_prober.go:29] interesting pod/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:15:57.356871 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:57.356763    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z" podUID="4e2d4e66-888d-455b-bded-5db746148762" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:15:57.782315 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:57.782280    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:57.788378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:57.788351    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:58.048113 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:58.048028    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:58.048981 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:58.048958    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:15:59.457829 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.457802    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:15:59.583267 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.583217    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-istio-envoy\") pod \"4e2d4e66-888d-455b-bded-5db746148762\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") "
Nov 03 00:15:59.583427 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.583277    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e2d4e66-888d-455b-bded-5db746148762-kube-api-access-dlxkj\" (UniqueName: \"kubernetes.io/projected/4e2d4e66-888d-455b-bded-5db746148762-kube-api-access-dlxkj\") pod \"4e2d4e66-888d-455b-bded-5db746148762\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") "
Nov 03 00:15:59.583427 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.583352    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-workload-certs\") pod \"4e2d4e66-888d-455b-bded-5db746148762\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") "
Nov 03 00:15:59.583427 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.583410    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e2d4e66-888d-455b-bded-5db746148762-istio-token\" (UniqueName: \"kubernetes.io/projected/4e2d4e66-888d-455b-bded-5db746148762-istio-token\") pod \"4e2d4e66-888d-455b-bded-5db746148762\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") "
Nov 03 00:15:59.583578 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.583460    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e2d4e66-888d-455b-bded-5db746148762-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e2d4e66-888d-455b-bded-5db746148762-istiod-ca-cert\") pod \"4e2d4e66-888d-455b-bded-5db746148762\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") "
Nov 03 00:15:59.583578 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.583549    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-workload-socket\") pod \"4e2d4e66-888d-455b-bded-5db746148762\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") "
Nov 03 00:15:59.583692 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.583611    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-istio-data\") pod \"4e2d4e66-888d-455b-bded-5db746148762\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") "
Nov 03 00:15:59.583745 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.583689    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e2d4e66-888d-455b-bded-5db746148762-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e2d4e66-888d-455b-bded-5db746148762-istio-podinfo\") pod \"4e2d4e66-888d-455b-bded-5db746148762\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") "
Nov 03 00:15:59.583745 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.583722    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-credential-socket\") pod \"4e2d4e66-888d-455b-bded-5db746148762\" (UID: \"4e2d4e66-888d-455b-bded-5db746148762\") "
Nov 03 00:15:59.584394 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.584122    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-workload-socket" pod "4e2d4e66-888d-455b-bded-5db746148762" (UID: "4e2d4e66-888d-455b-bded-5db746148762"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:15:59.584394 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.584337    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-credential-socket" pod "4e2d4e66-888d-455b-bded-5db746148762" (UID: "4e2d4e66-888d-455b-bded-5db746148762"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:15:59.584616 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.584575    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-istio-data" pod "4e2d4e66-888d-455b-bded-5db746148762" (UID: "4e2d4e66-888d-455b-bded-5db746148762"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:15:59.584744 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.584683    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2d4e66-888d-455b-bded-5db746148762-istiod-ca-cert" pod "4e2d4e66-888d-455b-bded-5db746148762" (UID: "4e2d4e66-888d-455b-bded-5db746148762"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:59.585179 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.585151    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-workload-certs" pod "4e2d4e66-888d-455b-bded-5db746148762" (UID: "4e2d4e66-888d-455b-bded-5db746148762"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:15:59.587217 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.587184    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2d4e66-888d-455b-bded-5db746148762-istio-token" pod "4e2d4e66-888d-455b-bded-5db746148762" (UID: "4e2d4e66-888d-455b-bded-5db746148762"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:59.587217 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.587206    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2d4e66-888d-455b-bded-5db746148762-kube-api-access-dlxkj" pod "4e2d4e66-888d-455b-bded-5db746148762" (UID: "4e2d4e66-888d-455b-bded-5db746148762"). InnerVolumeSpecName "kube-api-access-dlxkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:59.587803 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.587781    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-istio-envoy" pod "4e2d4e66-888d-455b-bded-5db746148762" (UID: "4e2d4e66-888d-455b-bded-5db746148762"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:15:59.587892 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.587850    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e2d4e66-888d-455b-bded-5db746148762-istio-podinfo" pod "4e2d4e66-888d-455b-bded-5db746148762" (UID: "4e2d4e66-888d-455b-bded-5db746148762"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:15:59.688024 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.685083    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:59.688024 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.685128    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e2d4e66-888d-455b-bded-5db746148762-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:59.688024 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.685142    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:59.688024 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.685156    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:59.688024 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.685170    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlxkj\" (UniqueName: \"kubernetes.io/projected/4e2d4e66-888d-455b-bded-5db746148762-kube-api-access-dlxkj\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:59.688024 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.685183    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:59.688024 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.685196    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e2d4e66-888d-455b-bded-5db746148762-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:59.688024 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.685208    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e2d4e66-888d-455b-bded-5db746148762-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:59.688024 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.685221    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2d4e66-888d-455b-bded-5db746148762-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:15:59.882973 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.882894    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb"
Nov 03 00:15:59.883166 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:15:59.883038    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw"
Nov 03 00:16:00.065133 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:00.065046    5012 generic.go:356] "Generic (PLEG): container finished" podID="4e2d4e66-888d-455b-bded-5db746148762" containerID="e7e865a3993f59f97722109eaf3735e86e6664844789f2db0bcbf98a26dfea42" exitCode=0
Nov 03 00:16:00.065133 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:00.065103    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"
Nov 03 00:16:00.065344 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:00.065128    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z" event={"ID":"4e2d4e66-888d-455b-bded-5db746148762","Type":"ContainerDied","Data":"e7e865a3993f59f97722109eaf3735e86e6664844789f2db0bcbf98a26dfea42"}
Nov 03 00:16:00.065344 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:00.065182    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z" event={"ID":"4e2d4e66-888d-455b-bded-5db746148762","Type":"ContainerDied","Data":"e82c30b837265791892392a42510332e46491af8f74410b84352cbb862423fed"}
Nov 03 00:16:00.065344 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:00.065207    5012 scope.go:122] "RemoveContainer" containerID="e7e865a3993f59f97722109eaf3735e86e6664844789f2db0bcbf98a26dfea42"
Nov 03 00:16:00.079208 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:00.079168    5012 scope.go:122] "RemoveContainer" containerID="9536692556bfaba6cb0d5ba03d940dc98589f328a3b7ea8ecd8317591603de30"
Nov 03 00:16:00.092779 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:00.092758    5012 scope.go:122] "RemoveContainer" containerID="e7e865a3993f59f97722109eaf3735e86e6664844789f2db0bcbf98a26dfea42"
Nov 03 00:16:00.093136 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:16:00.093105    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7e865a3993f59f97722109eaf3735e86e6664844789f2db0bcbf98a26dfea42\": container with ID starting with e7e865a3993f59f97722109eaf3735e86e6664844789f2db0bcbf98a26dfea42 not found: ID does not exist" containerID="e7e865a3993f59f97722109eaf3735e86e6664844789f2db0bcbf98a26dfea42"
Nov 03 00:16:00.093228 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:00.093146    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7e865a3993f59f97722109eaf3735e86e6664844789f2db0bcbf98a26dfea42"} err="failed to get container status \"e7e865a3993f59f97722109eaf3735e86e6664844789f2db0bcbf98a26dfea42\": rpc error: code = NotFound desc = could not find container \"e7e865a3993f59f97722109eaf3735e86e6664844789f2db0bcbf98a26dfea42\": container with ID starting with e7e865a3993f59f97722109eaf3735e86e6664844789f2db0bcbf98a26dfea42 not found: ID does not exist"
Nov 03 00:16:00.093228 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:00.093172    5012 scope.go:122] "RemoveContainer" containerID="9536692556bfaba6cb0d5ba03d940dc98589f328a3b7ea8ecd8317591603de30"
Nov 03 00:16:00.093495 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:16:00.093472    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9536692556bfaba6cb0d5ba03d940dc98589f328a3b7ea8ecd8317591603de30\": container with ID starting with 9536692556bfaba6cb0d5ba03d940dc98589f328a3b7ea8ecd8317591603de30 not found: ID does not exist" containerID="9536692556bfaba6cb0d5ba03d940dc98589f328a3b7ea8ecd8317591603de30"
Nov 03 00:16:00.093579 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:00.093501    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9536692556bfaba6cb0d5ba03d940dc98589f328a3b7ea8ecd8317591603de30"} err="failed to get container status \"9536692556bfaba6cb0d5ba03d940dc98589f328a3b7ea8ecd8317591603de30\": rpc error: code = NotFound desc = could not find container \"9536692556bfaba6cb0d5ba03d940dc98589f328a3b7ea8ecd8317591603de30\": container with ID starting with 9536692556bfaba6cb0d5ba03d940dc98589f328a3b7ea8ecd8317591603de30 not found: ID does not exist"
Nov 03 00:16:00.115955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:00.115931    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"]
Nov 03 00:16:00.121196 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:00.121173    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-e3e6ed37-openshift-default-8577b478b5-vts9z"]
Nov 03 00:16:01.073568 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:01.073500    5012 generic.go:356] "Generic (PLEG): container finished" podID="f474b6f6-e47d-4c13-8f63-b188ca20fce6" containerID="72608dab623518b5be1416a046137a2d785eabfa85fd43ea68eda7942a2737c4" exitCode=0
Nov 03 00:16:01.074080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:01.073572    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd" event={"ID":"f474b6f6-e47d-4c13-8f63-b188ca20fce6","Type":"ContainerDied","Data":"72608dab623518b5be1416a046137a2d785eabfa85fd43ea68eda7942a2737c4"}
Nov 03 00:16:01.074080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:01.074006    5012 scope.go:122] "RemoveContainer" containerID="72608dab623518b5be1416a046137a2d785eabfa85fd43ea68eda7942a2737c4"
Nov 03 00:16:01.075384 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:01.075356    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:16:01.075478 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:01.075431    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:16:01.075478 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:01.075465    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:16:01.075552 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:01.075491    5012 generic.go:356] "Generic (PLEG): container finished" podID="4233c829-fa1c-443e-b7b6-9522c977fa98" containerID="b41b4b89c96bab9fe680c62048ca2a8bf0d67e17835e735f5ac578d87d9a287f" exitCode=0
Nov 03 00:16:01.075595 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:01.075564    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6" event={"ID":"4233c829-fa1c-443e-b7b6-9522c977fa98","Type":"ContainerDied","Data":"b41b4b89c96bab9fe680c62048ca2a8bf0d67e17835e735f5ac578d87d9a287f"}
Nov 03 00:16:01.075985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:01.075935    5012 scope.go:122] "RemoveContainer" containerID="b41b4b89c96bab9fe680c62048ca2a8bf0d67e17835e735f5ac578d87d9a287f"
Nov 03 00:16:01.077109 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:01.077084    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:16:01.077192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:01.077144    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:16:01.077192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:01.077175    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:16:01.247932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:01.247210    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e2d4e66-888d-455b-bded-5db746148762" path="/var/lib/kubelet/pods/4e2d4e66-888d-455b-bded-5db746148762/volumes"
Nov 03 00:16:01.941789 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:01.941746    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td"]
Nov 03 00:16:01.942302 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:01.942287    5012 cpu_manager.go:414] "containerMap: removing container" podUID="4e2d4e66-888d-455b-bded-5db746148762" containerName="istio-proxy"
Nov 03 00:16:01.942350 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:01.942304    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e2d4e66-888d-455b-bded-5db746148762" containerName="istio-proxy"
Nov 03 00:16:01.942350 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:01.942329    5012 cpu_manager.go:414] "containerMap: removing container" podUID="4e2d4e66-888d-455b-bded-5db746148762" containerName="istio-proxy"
Nov 03 00:16:01.942350 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:01.942336    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e2d4e66-888d-455b-bded-5db746148762" containerName="istio-proxy"
Nov 03 00:16:01.942466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:01.942410    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e2d4e66-888d-455b-bded-5db746148762" containerName="istio-proxy"
Nov 03 00:16:01.945954 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:01.945939    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td"
Nov 03 00:16:01.967257 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:01.967235    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-b85343\""
Nov 03 00:16:01.970112 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:01.970093    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td"]
Nov 03 00:16:01.990195 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:01.990172    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-b85343-dockercfg-nrws8\""
Nov 03 00:16:02.041896 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.041855    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td\" (UID: \"ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td"
Nov 03 00:16:02.042097 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.041954    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxfc4\" (UniqueName: \"kubernetes.io/projected/ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e-kube-api-access-gxfc4\") pod \"payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td\" (UID: \"ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td"
Nov 03 00:16:02.050586 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.050555    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p"]
Nov 03 00:16:02.051144 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.051130    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e2d4e66-888d-455b-bded-5db746148762" containerName="istio-proxy"
Nov 03 00:16:02.054355 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.054340    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p"
Nov 03 00:16:02.074804 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.074778    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p"]
Nov 03 00:16:02.143576 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.143530    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td\" (UID: \"ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td"
Nov 03 00:16:02.152153 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.152117    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxfc4\" (UniqueName: \"kubernetes.io/projected/ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e-kube-api-access-gxfc4\") pod \"payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td\" (UID: \"ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td"
Nov 03 00:16:02.201450 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.144845    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td\" (UID: \"ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td"
Nov 03 00:16:02.252937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.249652    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxfc4\" (UniqueName: \"kubernetes.io/projected/ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e-kube-api-access-gxfc4\") pod \"payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td\" (UID: \"ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td"
Nov 03 00:16:02.252937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.249657    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd" event={"ID":"f474b6f6-e47d-4c13-8f63-b188ca20fce6","Type":"ContainerStarted","Data":"aa2ab1d62981a8870cc3945d8098f07f188d7eb45cb8dc8d0a6b72aca8b05304"}
Nov 03 00:16:02.252937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.252548    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6" event={"ID":"4233c829-fa1c-443e-b7b6-9522c977fa98","Type":"ContainerStarted","Data":"cbb0eb60aea42cde08a3cea874353b19329b0c5c040fd04228daa2bc0104009e"}
Nov 03 00:16:02.260081 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.260061    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td"
Nov 03 00:16:02.308747 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.307607    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm99m\" (UniqueName: \"kubernetes.io/projected/d7409925-56ce-416e-b0dc-2a4fde810094-kube-api-access-fm99m\") pod \"payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p\" (UID: \"d7409925-56ce-416e-b0dc-2a4fde810094\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p"
Nov 03 00:16:02.308747 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.307712    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7409925-56ce-416e-b0dc-2a4fde810094-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p\" (UID: \"d7409925-56ce-416e-b0dc-2a4fde810094\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p"
Nov 03 00:16:02.411934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.410756    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm99m\" (UniqueName: \"kubernetes.io/projected/d7409925-56ce-416e-b0dc-2a4fde810094-kube-api-access-fm99m\") pod \"payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p\" (UID: \"d7409925-56ce-416e-b0dc-2a4fde810094\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p"
Nov 03 00:16:02.411934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.410837    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7409925-56ce-416e-b0dc-2a4fde810094-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p\" (UID: \"d7409925-56ce-416e-b0dc-2a4fde810094\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p"
Nov 03 00:16:02.420937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.412891    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7409925-56ce-416e-b0dc-2a4fde810094-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p\" (UID: \"d7409925-56ce-416e-b0dc-2a4fde810094\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p"
Nov 03 00:16:02.442412 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.442361    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm99m\" (UniqueName: \"kubernetes.io/projected/d7409925-56ce-416e-b0dc-2a4fde810094-kube-api-access-fm99m\") pod \"payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p\" (UID: \"d7409925-56ce-416e-b0dc-2a4fde810094\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p"
Nov 03 00:16:02.643510 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.643482    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td"]
Nov 03 00:16:02.667613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.667589    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p"
Nov 03 00:16:02.667725 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.667645    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:16:02.667854 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.667831    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 00:16:02.667902 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.667877    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6" podUID="4233c829-fa1c-443e-b7b6-9522c977fa98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 00:16:02.780602 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.780576    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:16:02.780755 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.780729    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 03 00:16:02.780808 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.780786    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd" podUID="f474b6f6-e47d-4c13-8f63-b188ca20fce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 03 00:16:02.959726 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:02.959654    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957"
Nov 03 00:16:03.056943 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:16:03.056819    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7409925_56ce_416e_b0dc_2a4fde810094.slice/crio-3dcc34e41f182502d2ff48cb7136ac9000fac40896bd9bb5388128e6c53c9c96 WatchSource:0}: Error finding container 3dcc34e41f182502d2ff48cb7136ac9000fac40896bd9bb5388128e6c53c9c96: Status 404 returned error can't find the container with id 3dcc34e41f182502d2ff48cb7136ac9000fac40896bd9bb5388128e6c53c9c96
Nov 03 00:16:03.058170 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:03.057953    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p"]
Nov 03 00:16:03.262691 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:03.262610    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td" event={"ID":"ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e","Type":"ContainerStarted","Data":"3e10ec59a42f642736aab46f85569ab4e5b3a34aedb8f5a3ea337831f3619786"}
Nov 03 00:16:03.262691 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:03.262673    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td" event={"ID":"ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e","Type":"ContainerStarted","Data":"0c3737a318eda3495653db42fc2bc6e2c0506b4ba31525c3ee3eb5d49fa6c8ec"}
Nov 03 00:16:03.263179 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:03.262780    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td"
Nov 03 00:16:03.264290 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:03.264263    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p" event={"ID":"d7409925-56ce-416e-b0dc-2a4fde810094","Type":"ContainerStarted","Data":"226791cb1e23a28fca582c76c1c60dabbc3e10ed9d6cb7fb3d86635c0ff098b5"}
Nov 03 00:16:03.264430 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:03.264307    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p" event={"ID":"d7409925-56ce-416e-b0dc-2a4fde810094","Type":"ContainerStarted","Data":"3dcc34e41f182502d2ff48cb7136ac9000fac40896bd9bb5388128e6c53c9c96"}
Nov 03 00:16:03.264430 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:03.264388    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p"
Nov 03 00:16:03.331018 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:03.330956    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td" podStartSLOduration=2.330937306 podStartE2EDuration="2.330937306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:03.327372104 +0000 UTC m=+646.206956148" watchObservedRunningTime="2026-11-03 00:16:03.330937306 +0000 UTC m=+646.210521347"
Nov 03 00:16:03.365657 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:03.365606    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p" podStartSLOduration=1.365593979 podStartE2EDuration="1.365593979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:03.361602898 +0000 UTC m=+646.241186940" watchObservedRunningTime="2026-11-03 00:16:03.365593979 +0000 UTC m=+646.245178018"
Nov 03 00:16:03.668496 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:03.668458    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 00:16:03.668749 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:03.668725    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6" podUID="4233c829-fa1c-443e-b7b6-9522c977fa98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 00:16:03.780788 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:03.780753    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 03 00:16:03.780981 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:03.780809    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd" podUID="f474b6f6-e47d-4c13-8f63-b188ca20fce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 03 00:16:04.668299 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:04.668267    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 00:16:04.668758 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:04.668728    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6" podUID="4233c829-fa1c-443e-b7b6-9522c977fa98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 00:16:04.781370 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:04.781335    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 03 00:16:04.781566 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:04.781397    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd" podUID="f474b6f6-e47d-4c13-8f63-b188ca20fce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 03 00:16:05.668070 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:05.668041    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 00:16:05.668297 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:05.668275    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6" podUID="4233c829-fa1c-443e-b7b6-9522c977fa98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 00:16:05.780801 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:05.780762    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 03 00:16:05.781205 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:05.780826    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd" podUID="f474b6f6-e47d-4c13-8f63-b188ca20fce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 03 00:16:06.668797 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:06.668761    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 00:16:06.668974 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:06.668842    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6" podUID="4233c829-fa1c-443e-b7b6-9522c977fa98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 00:16:06.780565 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:06.780523    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 03 00:16:06.780733 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:06.780586    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd" podUID="f474b6f6-e47d-4c13-8f63-b188ca20fce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 03 00:16:07.394155 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:16:07.394115    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:16:07.672797 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:07.672707    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:16:07.673168 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:07.673146    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:16:07.673774 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:07.673751    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 00:16:07.785015 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:07.784965    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:16:07.785397 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:07.785377    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:16:07.786105 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:07.786082    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 00:16:12.390374 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.390340    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"]
Nov 03 00:16:12.397261 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.397233    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:16:12.413926 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.413890    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"]
Nov 03 00:16:12.420657 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.420602    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8e90c5\"/\"kube-root-ca.crt\""
Nov 03 00:16:12.420657 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.420617    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8e90c5\"/\"openshift-service-ca.crt\""
Nov 03 00:16:12.421462 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.421443    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8e90c5\"/\"default-dockercfg-79f5r\""
Nov 03 00:16:12.421620 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.421485    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8e90c5\"/\"simulator-w3-8e90c5-kserve-self-signed-certs\""
Nov 03 00:16:12.505032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.505004    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-tmp-dir\") pod \"simulator-w3-8e90c5-kserve-5b47b97984-kbqwt\" (UID: \"170314c3-bfce-424b-bba2-3dc03d1b80e4\") " pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:16:12.505170 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.505062    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4gq9\" (UniqueName: \"kubernetes.io/projected/170314c3-bfce-424b-bba2-3dc03d1b80e4-kube-api-access-b4gq9\") pod \"simulator-w3-8e90c5-kserve-5b47b97984-kbqwt\" (UID: \"170314c3-bfce-424b-bba2-3dc03d1b80e4\") " pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:16:12.505170 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.505142    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-home\") pod \"simulator-w3-8e90c5-kserve-5b47b97984-kbqwt\" (UID: \"170314c3-bfce-424b-bba2-3dc03d1b80e4\") " pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:16:12.505275 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.505192    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-model-cache\") pod \"simulator-w3-8e90c5-kserve-5b47b97984-kbqwt\" (UID: \"170314c3-bfce-424b-bba2-3dc03d1b80e4\") " pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:16:12.505314 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.505297    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/170314c3-bfce-424b-bba2-3dc03d1b80e4-tls-certs\") pod \"simulator-w3-8e90c5-kserve-5b47b97984-kbqwt\" (UID: \"170314c3-bfce-424b-bba2-3dc03d1b80e4\") " pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:16:12.505387 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.505362    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-dshm\") pod \"simulator-w3-8e90c5-kserve-5b47b97984-kbqwt\" (UID: \"170314c3-bfce-424b-bba2-3dc03d1b80e4\") " pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:16:12.606171 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.606146    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4gq9\" (UniqueName: \"kubernetes.io/projected/170314c3-bfce-424b-bba2-3dc03d1b80e4-kube-api-access-b4gq9\") pod \"simulator-w3-8e90c5-kserve-5b47b97984-kbqwt\" (UID: \"170314c3-bfce-424b-bba2-3dc03d1b80e4\") " pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:16:12.606282 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.606184    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-home\") pod \"simulator-w3-8e90c5-kserve-5b47b97984-kbqwt\" (UID: \"170314c3-bfce-424b-bba2-3dc03d1b80e4\") " pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:16:12.606282 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.606227    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-model-cache\") pod \"simulator-w3-8e90c5-kserve-5b47b97984-kbqwt\" (UID: \"170314c3-bfce-424b-bba2-3dc03d1b80e4\") " pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:16:12.606282 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.606256    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/170314c3-bfce-424b-bba2-3dc03d1b80e4-tls-certs\") pod \"simulator-w3-8e90c5-kserve-5b47b97984-kbqwt\" (UID: \"170314c3-bfce-424b-bba2-3dc03d1b80e4\") " pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:16:12.606430 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.606408    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-dshm\") pod \"simulator-w3-8e90c5-kserve-5b47b97984-kbqwt\" (UID: \"170314c3-bfce-424b-bba2-3dc03d1b80e4\") " pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:16:12.606552 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.606453    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-tmp-dir\") pod \"simulator-w3-8e90c5-kserve-5b47b97984-kbqwt\" (UID: \"170314c3-bfce-424b-bba2-3dc03d1b80e4\") " pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:16:12.606654 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.606634    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-home\") pod \"simulator-w3-8e90c5-kserve-5b47b97984-kbqwt\" (UID: \"170314c3-bfce-424b-bba2-3dc03d1b80e4\") " pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:16:12.606712 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.606675    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-model-cache\") pod \"simulator-w3-8e90c5-kserve-5b47b97984-kbqwt\" (UID: \"170314c3-bfce-424b-bba2-3dc03d1b80e4\") " pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:16:12.606813 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.606796    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-tmp-dir\") pod \"simulator-w3-8e90c5-kserve-5b47b97984-kbqwt\" (UID: \"170314c3-bfce-424b-bba2-3dc03d1b80e4\") " pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:16:12.608521 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.608500    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-dshm\") pod \"simulator-w3-8e90c5-kserve-5b47b97984-kbqwt\" (UID: \"170314c3-bfce-424b-bba2-3dc03d1b80e4\") " pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:16:12.608779 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.608763    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/170314c3-bfce-424b-bba2-3dc03d1b80e4-tls-certs\") pod \"simulator-w3-8e90c5-kserve-5b47b97984-kbqwt\" (UID: \"170314c3-bfce-424b-bba2-3dc03d1b80e4\") " pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:16:12.626597 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.626576    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4gq9\" (UniqueName: \"kubernetes.io/projected/170314c3-bfce-424b-bba2-3dc03d1b80e4-kube-api-access-b4gq9\") pod \"simulator-w3-8e90c5-kserve-5b47b97984-kbqwt\" (UID: \"170314c3-bfce-424b-bba2-3dc03d1b80e4\") " pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:16:12.713921 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:12.713854    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:16:13.040134 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:13.040106    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"]
Nov 03 00:16:13.041290 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:16:13.041264    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod170314c3_bfce_424b_bba2_3dc03d1b80e4.slice/crio-b4f2fb243d0a08eef24c64ee32fb84d31a1d9201728102c22477d71babc54203 WatchSource:0}: Error finding container b4f2fb243d0a08eef24c64ee32fb84d31a1d9201728102c22477d71babc54203: Status 404 returned error can't find the container with id b4f2fb243d0a08eef24c64ee32fb84d31a1d9201728102c22477d71babc54203
Nov 03 00:16:13.336241 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:13.336164    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" event={"ID":"170314c3-bfce-424b-bba2-3dc03d1b80e4","Type":"ContainerStarted","Data":"b431696cdbece4beff53510320a1491d5117615f770679c968ba920a50430a5b"}
Nov 03 00:16:13.336241 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:13.336209    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" event={"ID":"170314c3-bfce-424b-bba2-3dc03d1b80e4","Type":"ContainerStarted","Data":"b4f2fb243d0a08eef24c64ee32fb84d31a1d9201728102c22477d71babc54203"}
Nov 03 00:16:13.405369 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:13.405322    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podStartSLOduration=1.405310016 podStartE2EDuration="1.405310016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:13.402069486 +0000 UTC m=+656.281653528" watchObservedRunningTime="2026-11-03 00:16:13.405310016 +0000 UTC m=+656.284894058"
Nov 03 00:16:13.714392 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:13.714355    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:16:14.274421 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:14.274390    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p"
Nov 03 00:16:14.274593 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:14.274569    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td"
Nov 03 00:16:16.360803 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:16.360770    5012 generic.go:356] "Generic (PLEG): container finished" podID="8a8544a5-55b2-4894-a924-6c408585fe18" containerID="d2305442074b0e55e4f9166d70eac2b23ad7f385503cbd9b9e2e127e34a1347c" exitCode=1
Nov 03 00:16:16.361304 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:16.360819    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wz9nt" event={"ID":"8a8544a5-55b2-4894-a924-6c408585fe18","Type":"ContainerDied","Data":"d2305442074b0e55e4f9166d70eac2b23ad7f385503cbd9b9e2e127e34a1347c"}
Nov 03 00:16:16.361304 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:16.361199    5012 scope.go:122] "RemoveContainer" containerID="d2305442074b0e55e4f9166d70eac2b23ad7f385503cbd9b9e2e127e34a1347c"
Nov 03 00:16:16.373194 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:16.373174    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 03 00:16:17.371087 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:17.371055    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wz9nt" event={"ID":"8a8544a5-55b2-4894-a924-6c408585fe18","Type":"ContainerStarted","Data":"543b761002be30022176dc41f575037dd767dd193f1d8e7326ddad4347cf54f9"}
Nov 03 00:16:17.371558 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:17.371281    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wz9nt"
Nov 03 00:16:22.714654 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:22.714614    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:16:22.727277 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:22.727253    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:16:23.420248 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:23.420213    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:16:28.380214 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:28.380140    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wz9nt"
Nov 03 00:16:37.362486 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:16:37.362452    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:16:59.090091 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:59.090053    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:16:59.090511 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:16:59.090142    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9" podUID="8a090865-de87-442c-806a-76e91d5d374c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:17:00.683128 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:00.683092    5012 generic.go:356] "Generic (PLEG): container finished" podID="8a090865-de87-442c-806a-76e91d5d374c" containerID="c910e4fab82b248bac75a840eca104a7ab622e2d9564d41f36ffbc91848e546a" exitCode=0
Nov 03 00:17:00.683546 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:00.683167    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9" event={"ID":"8a090865-de87-442c-806a-76e91d5d374c","Type":"ContainerDied","Data":"c910e4fab82b248bac75a840eca104a7ab622e2d9564d41f36ffbc91848e546a"}
Nov 03 00:17:00.683588 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:00.683545    5012 scope.go:122] "RemoveContainer" containerID="c910e4fab82b248bac75a840eca104a7ab622e2d9564d41f36ffbc91848e546a"
Nov 03 00:17:00.684854 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:00.684827    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:17:00.684955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:00.684895    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:17:00.684998 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:00.684952    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:17:01.713846 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:01.713804    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9" event={"ID":"8a090865-de87-442c-806a-76e91d5d374c","Type":"ContainerStarted","Data":"dd014bbb64d8d2bff6703fbf3b4e5022dd27c5703717c686e1c67310e75023ae"}
Nov 03 00:17:01.782494 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:01.782462    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:17:01.782711 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:01.782686    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 00:17:01.782774 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:01.782754    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9" podUID="8a090865-de87-442c-806a-76e91d5d374c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 00:17:02.781972 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:02.781943    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 00:17:02.782378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:02.781989    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9" podUID="8a090865-de87-442c-806a-76e91d5d374c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 00:17:03.782350 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:03.782315    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 00:17:03.782757 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:03.782365    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9" podUID="8a090865-de87-442c-806a-76e91d5d374c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 00:17:04.781836 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:04.781794    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 00:17:04.782056 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:04.781855    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9" podUID="8a090865-de87-442c-806a-76e91d5d374c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 00:17:05.782368 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:05.782334    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 00:17:05.782769 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:05.782390    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9" podUID="8a090865-de87-442c-806a-76e91d5d374c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 00:17:06.782804 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:06.782769    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 00:17:06.783256 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:06.782829    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9" podUID="8a090865-de87-442c-806a-76e91d5d374c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 00:17:07.362779 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:17:07.362740    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:17:07.785813 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:07.785737    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:17:07.786377 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:07.786215    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:17:07.786876 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:07.786856    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:17:27.663303 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.663260    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"]
Nov 03 00:17:27.668550 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.668523    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:17:27.685744 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.685710    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"]
Nov 03 00:17:27.691438 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.691406    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b85343\"/\"kube-root-ca.crt\""
Nov 03 00:17:27.695047 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.695015    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b85343\"/\"default-dockercfg-hzjrw\""
Nov 03 00:17:27.695047 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.695017    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b85343\"/\"openshift-service-ca.crt\""
Nov 03 00:17:27.695290 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.695243    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b85343\"/\"premium-w1-b85343-kserve-self-signed-certs\""
Nov 03 00:17:27.794442 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.794403    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-tmp-dir\") pod \"premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv\" (UID: \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\") " pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:17:27.794618 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.794467    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-home\") pod \"premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv\" (UID: \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\") " pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:17:27.794618 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.794559    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-dshm\") pod \"premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv\" (UID: \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\") " pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:17:27.794618 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.794587    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-model-cache\") pod \"premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv\" (UID: \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\") " pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:17:27.794618 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.794609    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-698zz\" (UniqueName: \"kubernetes.io/projected/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-kube-api-access-698zz\") pod \"premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv\" (UID: \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\") " pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:17:27.794758 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.794682    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-tls-certs\") pod \"premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv\" (UID: \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\") " pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:17:27.895577 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.895533    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-home\") pod \"premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv\" (UID: \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\") " pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:17:27.895759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.895626    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-dshm\") pod \"premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv\" (UID: \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\") " pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:17:27.895759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.895662    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-model-cache\") pod \"premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv\" (UID: \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\") " pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:17:27.895759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.895692    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-698zz\" (UniqueName: \"kubernetes.io/projected/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-kube-api-access-698zz\") pod \"premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv\" (UID: \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\") " pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:17:27.895759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.895728    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-tls-certs\") pod \"premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv\" (UID: \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\") " pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:17:27.895975 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.895848    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-tmp-dir\") pod \"premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv\" (UID: \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\") " pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:17:27.896081 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.896053    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-home\") pod \"premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv\" (UID: \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\") " pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:17:27.896133 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.896085    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-model-cache\") pod \"premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv\" (UID: \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\") " pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:17:27.896251 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.896230    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-tmp-dir\") pod \"premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv\" (UID: \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\") " pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:17:27.898046 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.898022    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-dshm\") pod \"premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv\" (UID: \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\") " pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:17:27.898355 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.898334    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-tls-certs\") pod \"premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv\" (UID: \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\") " pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:17:27.912727 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.912689    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-698zz\" (UniqueName: \"kubernetes.io/projected/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-kube-api-access-698zz\") pod \"premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv\" (UID: \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\") " pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:17:27.984885 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:27.984796    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:17:28.287031 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:28.286990    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"]
Nov 03 00:17:28.288795 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:17:28.288749    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba41fa63_4d99_49a5_aa53_5b15efddb4a5.slice/crio-8d8bab76cd9c6f968d2114c2311660547bc70b913c2bd1ca46c7f1eb4ffe79a9 WatchSource:0}: Error finding container 8d8bab76cd9c6f968d2114c2311660547bc70b913c2bd1ca46c7f1eb4ffe79a9: Status 404 returned error can't find the container with id 8d8bab76cd9c6f968d2114c2311660547bc70b913c2bd1ca46c7f1eb4ffe79a9
Nov 03 00:17:28.905809 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:28.905769    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" event={"ID":"ba41fa63-4d99-49a5-aa53-5b15efddb4a5","Type":"ContainerStarted","Data":"a175a22f87acfac139b9e246b43af2f1da0354dde5232c86e1395002d3fabe03"}
Nov 03 00:17:28.905809 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:28.905822    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" event={"ID":"ba41fa63-4d99-49a5-aa53-5b15efddb4a5","Type":"ContainerStarted","Data":"8d8bab76cd9c6f968d2114c2311660547bc70b913c2bd1ca46c7f1eb4ffe79a9"}
Nov 03 00:17:28.939880 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:28.939820    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podStartSLOduration=1.9398054230000001 podStartE2EDuration="1.939805423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:17:28.938984183 +0000 UTC m=+731.818568225" watchObservedRunningTime="2026-11-03 00:17:28.939805423 +0000 UTC m=+731.819389502"
Nov 03 00:17:28.985985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:28.985904    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:17:37.366965 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:17:37.366904    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:17:37.985839 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:37.985804    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:17:38.015501 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:38.015202    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:17:38.990136 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:38.990108    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:17:56.944123 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:56.944088    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ae896ef-r56pz"]
Nov 03 00:17:56.948252 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:56.948234    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae896ef-r56pz"
Nov 03 00:17:56.970828 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:56.970797    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ae896ef-r56pz"]
Nov 03 00:17:56.984385 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:56.971223    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 00:17:56.984530 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:56.972402    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:17:56.984530 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:56.973696    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:17:56.984530 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:56.974201    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 00:17:57.029404 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:57.029374    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqkrm\" (UniqueName: \"kubernetes.io/projected/1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1-kube-api-access-dqkrm\") pod \"maas-api-revoke-sub-5ae896ef-r56pz\" (UID: \"1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae896ef-r56pz"
Nov 03 00:17:57.029543 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:57.029412    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ae896ef-r56pz\" (UID: \"1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae896ef-r56pz"
Nov 03 00:17:57.029543 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:57.029526    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ae896ef-r56pz\" (UID: \"1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae896ef-r56pz"
Nov 03 00:17:57.130815 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:57.130775    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqkrm\" (UniqueName: \"kubernetes.io/projected/1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1-kube-api-access-dqkrm\") pod \"maas-api-revoke-sub-5ae896ef-r56pz\" (UID: \"1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae896ef-r56pz"
Nov 03 00:17:57.131048 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:57.130827    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ae896ef-r56pz\" (UID: \"1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae896ef-r56pz"
Nov 03 00:17:57.131048 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:57.130901    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ae896ef-r56pz\" (UID: \"1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae896ef-r56pz"
Nov 03 00:17:57.131491 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:57.131467    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ae896ef-r56pz\" (UID: \"1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae896ef-r56pz"
Nov 03 00:17:57.133843 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:57.133817    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ae896ef-r56pz\" (UID: \"1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae896ef-r56pz"
Nov 03 00:17:57.147465 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:57.147436    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqkrm\" (UniqueName: \"kubernetes.io/projected/1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1-kube-api-access-dqkrm\") pod \"maas-api-revoke-sub-5ae896ef-r56pz\" (UID: \"1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae896ef-r56pz"
Nov 03 00:17:57.290619 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:57.290530    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae896ef-r56pz"
Nov 03 00:17:57.593068 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:57.593036    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ae896ef-r56pz"]
Nov 03 00:17:57.594107 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:17:57.594080    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eeebb6b_69fe_4eb9_b182_c8a6f12c05f1.slice/crio-f80c32420dc97b30415260670b4533a97e0c858b2007477e937d6f0ae64c8fba WatchSource:0}: Error finding container f80c32420dc97b30415260670b4533a97e0c858b2007477e937d6f0ae64c8fba: Status 404 returned error can't find the container with id f80c32420dc97b30415260670b4533a97e0c858b2007477e937d6f0ae64c8fba
Nov 03 00:17:58.119373 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:58.119327    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae896ef-r56pz" event={"ID":"1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1","Type":"ContainerStarted","Data":"bc4634160c87716f50e46ae36505894c76b29e0f60a0c8d0c455f596b2084d54"}
Nov 03 00:17:58.119836 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:58.119396    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae896ef-r56pz" event={"ID":"1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1","Type":"ContainerStarted","Data":"f80c32420dc97b30415260670b4533a97e0c858b2007477e937d6f0ae64c8fba"}
Nov 03 00:17:58.154226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:58.154123    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae896ef-r56pz" podStartSLOduration=2.154104997 podStartE2EDuration="2.154104997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:17:58.151558674 +0000 UTC m=+761.031142718" watchObservedRunningTime="2026-11-03 00:17:58.154104997 +0000 UTC m=+761.033689040"
Nov 03 00:17:59.129509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:59.129473    5012 generic.go:356] "Generic (PLEG): container finished" podID="1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1" containerID="bc4634160c87716f50e46ae36505894c76b29e0f60a0c8d0c455f596b2084d54" exitCode=0
Nov 03 00:17:59.129977 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:17:59.129560    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae896ef-r56pz" event={"ID":"1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1","Type":"ContainerDied","Data":"bc4634160c87716f50e46ae36505894c76b29e0f60a0c8d0c455f596b2084d54"}
Nov 03 00:18:00.295384 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:00.295361    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae896ef-r56pz"
Nov 03 00:18:00.481898 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:00.481810    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1-kube-api-access-dqkrm\" (UniqueName: \"kubernetes.io/projected/1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1-kube-api-access-dqkrm\") pod \"1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1\" (UID: \"1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1\") "
Nov 03 00:18:00.481898 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:00.481854    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1-maas-api-openshift-ca\") pod \"1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1\" (UID: \"1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1\") "
Nov 03 00:18:00.481898 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:00.481879    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1-maas-api-xks-ca\") pod \"1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1\" (UID: \"1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1\") "
Nov 03 00:18:00.482366 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:00.482334    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1-maas-api-openshift-ca" pod "1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1" (UID: "1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:18:00.484108 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:00.484083    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1-kube-api-access-dqkrm" pod "1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1" (UID: "1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1"). InnerVolumeSpecName "kube-api-access-dqkrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:18:00.484265 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:00.484245    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1-maas-api-xks-ca" pod "1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1" (UID: "1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:18:00.583491 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:00.583452    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqkrm\" (UniqueName: \"kubernetes.io/projected/1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1-kube-api-access-dqkrm\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:18:00.583491 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:00.583489    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:18:00.583680 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:00.583503    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:18:01.148229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:01.148199    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae896ef-r56pz"
Nov 03 00:18:01.148229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:01.148224    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae896ef-r56pz" event={"ID":"1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1","Type":"ContainerDied","Data":"f80c32420dc97b30415260670b4533a97e0c858b2007477e937d6f0ae64c8fba"}
Nov 03 00:18:01.148437 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:01.148304    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f80c32420dc97b30415260670b4533a97e0c858b2007477e937d6f0ae64c8fba"
Nov 03 00:18:03.842501 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:03.842459    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98453474-stnhm"]
Nov 03 00:18:03.843141 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:03.843012    5012 cpu_manager.go:414] "containerMap: removing container" podUID="1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1" containerName="revoke-keys"
Nov 03 00:18:03.843141 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:03.843025    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1" containerName="revoke-keys"
Nov 03 00:18:03.843141 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:03.843109    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1" containerName="revoke-keys"
Nov 03 00:18:03.846456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:03.846437    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98453474-stnhm"
Nov 03 00:18:03.864466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:03.864436    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98453474-stnhm"]
Nov 03 00:18:03.869235 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:03.869209    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:18:03.869344 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:03.869312    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:18:03.918375 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:03.918335    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e714a7c2-61b0-4419-a5bb-f09fb5336643-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98453474-stnhm\" (UID: \"e714a7c2-61b0-4419-a5bb-f09fb5336643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98453474-stnhm"
Nov 03 00:18:03.918582 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:03.918451    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqf6z\" (UniqueName: \"kubernetes.io/projected/e714a7c2-61b0-4419-a5bb-f09fb5336643-kube-api-access-wqf6z\") pod \"maas-api-revoke-sub-98453474-stnhm\" (UID: \"e714a7c2-61b0-4419-a5bb-f09fb5336643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98453474-stnhm"
Nov 03 00:18:03.918582 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:03.918503    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e714a7c2-61b0-4419-a5bb-f09fb5336643-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98453474-stnhm\" (UID: \"e714a7c2-61b0-4419-a5bb-f09fb5336643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98453474-stnhm"
Nov 03 00:18:03.920164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:03.920134    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 00:18:03.920164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:03.920139    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 00:18:04.019120 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:04.019090    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e714a7c2-61b0-4419-a5bb-f09fb5336643-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98453474-stnhm\" (UID: \"e714a7c2-61b0-4419-a5bb-f09fb5336643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98453474-stnhm"
Nov 03 00:18:04.019287 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:04.019166    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqf6z\" (UniqueName: \"kubernetes.io/projected/e714a7c2-61b0-4419-a5bb-f09fb5336643-kube-api-access-wqf6z\") pod \"maas-api-revoke-sub-98453474-stnhm\" (UID: \"e714a7c2-61b0-4419-a5bb-f09fb5336643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98453474-stnhm"
Nov 03 00:18:04.019287 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:04.019188    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e714a7c2-61b0-4419-a5bb-f09fb5336643-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98453474-stnhm\" (UID: \"e714a7c2-61b0-4419-a5bb-f09fb5336643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98453474-stnhm"
Nov 03 00:18:04.019866 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:04.019834    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e714a7c2-61b0-4419-a5bb-f09fb5336643-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98453474-stnhm\" (UID: \"e714a7c2-61b0-4419-a5bb-f09fb5336643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98453474-stnhm"
Nov 03 00:18:04.021545 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:04.021523    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e714a7c2-61b0-4419-a5bb-f09fb5336643-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98453474-stnhm\" (UID: \"e714a7c2-61b0-4419-a5bb-f09fb5336643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98453474-stnhm"
Nov 03 00:18:04.035678 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:04.035654    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqf6z\" (UniqueName: \"kubernetes.io/projected/e714a7c2-61b0-4419-a5bb-f09fb5336643-kube-api-access-wqf6z\") pod \"maas-api-revoke-sub-98453474-stnhm\" (UID: \"e714a7c2-61b0-4419-a5bb-f09fb5336643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98453474-stnhm"
Nov 03 00:18:04.161521 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:04.161480    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98453474-stnhm"
Nov 03 00:18:04.464675 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:18:04.464643    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode714a7c2_61b0_4419_a5bb_f09fb5336643.slice/crio-dbf22b6c9277df01df92aa3639a83f3c4d9af8a0d7fdd09d3bbdfdb06517ce5c WatchSource:0}: Error finding container dbf22b6c9277df01df92aa3639a83f3c4d9af8a0d7fdd09d3bbdfdb06517ce5c: Status 404 returned error can't find the container with id dbf22b6c9277df01df92aa3639a83f3c4d9af8a0d7fdd09d3bbdfdb06517ce5c
Nov 03 00:18:04.465122 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:04.465097    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98453474-stnhm"]
Nov 03 00:18:05.176585 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:05.176502    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98453474-stnhm" event={"ID":"e714a7c2-61b0-4419-a5bb-f09fb5336643","Type":"ContainerStarted","Data":"73b444ceeb3942b6d029d688e8d2dade204984165456588a9316b03d19f160ed"}
Nov 03 00:18:05.176585 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:05.176556    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98453474-stnhm" event={"ID":"e714a7c2-61b0-4419-a5bb-f09fb5336643","Type":"ContainerStarted","Data":"dbf22b6c9277df01df92aa3639a83f3c4d9af8a0d7fdd09d3bbdfdb06517ce5c"}
Nov 03 00:18:05.210805 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:05.210757    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98453474-stnhm" podStartSLOduration=2.210744533 podStartE2EDuration="2.210744533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:05.208018915 +0000 UTC m=+768.087602958" watchObservedRunningTime="2026-11-03 00:18:05.210744533 +0000 UTC m=+768.090328576"
Nov 03 00:18:06.184535 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:06.184501    5012 generic.go:356] "Generic (PLEG): container finished" podID="e714a7c2-61b0-4419-a5bb-f09fb5336643" containerID="73b444ceeb3942b6d029d688e8d2dade204984165456588a9316b03d19f160ed" exitCode=0
Nov 03 00:18:06.184967 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:06.184591    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98453474-stnhm" event={"ID":"e714a7c2-61b0-4419-a5bb-f09fb5336643","Type":"ContainerDied","Data":"73b444ceeb3942b6d029d688e8d2dade204984165456588a9316b03d19f160ed"}
Nov 03 00:18:07.335892 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:07.335869    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98453474-stnhm"
Nov 03 00:18:07.361959 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:18:07.361926    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:18:07.458611 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:07.458515    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e714a7c2-61b0-4419-a5bb-f09fb5336643-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e714a7c2-61b0-4419-a5bb-f09fb5336643-maas-api-openshift-ca\") pod \"e714a7c2-61b0-4419-a5bb-f09fb5336643\" (UID: \"e714a7c2-61b0-4419-a5bb-f09fb5336643\") "
Nov 03 00:18:07.458802 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:07.458633    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e714a7c2-61b0-4419-a5bb-f09fb5336643-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e714a7c2-61b0-4419-a5bb-f09fb5336643-maas-api-xks-ca\") pod \"e714a7c2-61b0-4419-a5bb-f09fb5336643\" (UID: \"e714a7c2-61b0-4419-a5bb-f09fb5336643\") "
Nov 03 00:18:07.458802 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:07.458676    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e714a7c2-61b0-4419-a5bb-f09fb5336643-kube-api-access-wqf6z\" (UniqueName: \"kubernetes.io/projected/e714a7c2-61b0-4419-a5bb-f09fb5336643-kube-api-access-wqf6z\") pod \"e714a7c2-61b0-4419-a5bb-f09fb5336643\" (UID: \"e714a7c2-61b0-4419-a5bb-f09fb5336643\") "
Nov 03 00:18:07.458938 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:07.458889    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e714a7c2-61b0-4419-a5bb-f09fb5336643-maas-api-openshift-ca" pod "e714a7c2-61b0-4419-a5bb-f09fb5336643" (UID: "e714a7c2-61b0-4419-a5bb-f09fb5336643"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:18:07.459082 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:07.459064    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e714a7c2-61b0-4419-a5bb-f09fb5336643-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:18:07.460896 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:07.460869    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e714a7c2-61b0-4419-a5bb-f09fb5336643-maas-api-xks-ca" pod "e714a7c2-61b0-4419-a5bb-f09fb5336643" (UID: "e714a7c2-61b0-4419-a5bb-f09fb5336643"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:18:07.461037 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:07.460930    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e714a7c2-61b0-4419-a5bb-f09fb5336643-kube-api-access-wqf6z" pod "e714a7c2-61b0-4419-a5bb-f09fb5336643" (UID: "e714a7c2-61b0-4419-a5bb-f09fb5336643"). InnerVolumeSpecName "kube-api-access-wqf6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:18:07.559745 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:07.559718    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e714a7c2-61b0-4419-a5bb-f09fb5336643-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:18:07.559745 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:07.559744    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqf6z\" (UniqueName: \"kubernetes.io/projected/e714a7c2-61b0-4419-a5bb-f09fb5336643-kube-api-access-wqf6z\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:18:08.202288 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:08.202260    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98453474-stnhm"
Nov 03 00:18:08.202461 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:08.202295    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98453474-stnhm" event={"ID":"e714a7c2-61b0-4419-a5bb-f09fb5336643","Type":"ContainerDied","Data":"dbf22b6c9277df01df92aa3639a83f3c4d9af8a0d7fdd09d3bbdfdb06517ce5c"}
Nov 03 00:18:08.202461 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:08.202342    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbf22b6c9277df01df92aa3639a83f3c4d9af8a0d7fdd09d3bbdfdb06517ce5c"
Nov 03 00:18:09.134993 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:09.134957    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ccd8e17-qvkk5"]
Nov 03 00:18:09.135519 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:09.135503    5012 cpu_manager.go:414] "containerMap: removing container" podUID="e714a7c2-61b0-4419-a5bb-f09fb5336643" containerName="revoke-keys"
Nov 03 00:18:09.135558 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:09.135519    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e714a7c2-61b0-4419-a5bb-f09fb5336643" containerName="revoke-keys"
Nov 03 00:18:09.135630 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:09.135620    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="e714a7c2-61b0-4419-a5bb-f09fb5336643" containerName="revoke-keys"
Nov 03 00:18:09.139366 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:09.139349    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccd8e17-qvkk5"
Nov 03 00:18:09.156593 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:09.156568    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ccd8e17-qvkk5"]
Nov 03 00:18:09.164595 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:09.164571    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:18:09.164758 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:09.164733    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 00:18:09.164758 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:09.164663    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 00:18:09.164896 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:09.164663    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:18:09.276107 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:09.276075    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng6rt\" (UniqueName: \"kubernetes.io/projected/ca01721d-5c8f-4127-977f-5fa5eaf3100e-kube-api-access-ng6rt\") pod \"maas-api-revoke-sub-7ccd8e17-qvkk5\" (UID: \"ca01721d-5c8f-4127-977f-5fa5eaf3100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccd8e17-qvkk5"
Nov 03 00:18:09.276270 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:09.276129    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca01721d-5c8f-4127-977f-5fa5eaf3100e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ccd8e17-qvkk5\" (UID: \"ca01721d-5c8f-4127-977f-5fa5eaf3100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccd8e17-qvkk5"
Nov 03 00:18:09.276270 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:09.276237    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca01721d-5c8f-4127-977f-5fa5eaf3100e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ccd8e17-qvkk5\" (UID: \"ca01721d-5c8f-4127-977f-5fa5eaf3100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccd8e17-qvkk5"
Nov 03 00:18:09.377078 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:09.377046    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca01721d-5c8f-4127-977f-5fa5eaf3100e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ccd8e17-qvkk5\" (UID: \"ca01721d-5c8f-4127-977f-5fa5eaf3100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccd8e17-qvkk5"
Nov 03 00:18:09.377264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:09.377211    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng6rt\" (UniqueName: \"kubernetes.io/projected/ca01721d-5c8f-4127-977f-5fa5eaf3100e-kube-api-access-ng6rt\") pod \"maas-api-revoke-sub-7ccd8e17-qvkk5\" (UID: \"ca01721d-5c8f-4127-977f-5fa5eaf3100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccd8e17-qvkk5"
Nov 03 00:18:09.377607 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:09.377582    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca01721d-5c8f-4127-977f-5fa5eaf3100e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ccd8e17-qvkk5\" (UID: \"ca01721d-5c8f-4127-977f-5fa5eaf3100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccd8e17-qvkk5"
Nov 03 00:18:09.377899 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:09.377872    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca01721d-5c8f-4127-977f-5fa5eaf3100e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ccd8e17-qvkk5\" (UID: \"ca01721d-5c8f-4127-977f-5fa5eaf3100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccd8e17-qvkk5"
Nov 03 00:18:09.383316 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:09.383284    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca01721d-5c8f-4127-977f-5fa5eaf3100e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ccd8e17-qvkk5\" (UID: \"ca01721d-5c8f-4127-977f-5fa5eaf3100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccd8e17-qvkk5"
Nov 03 00:18:09.393366 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:09.393316    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng6rt\" (UniqueName: \"kubernetes.io/projected/ca01721d-5c8f-4127-977f-5fa5eaf3100e-kube-api-access-ng6rt\") pod \"maas-api-revoke-sub-7ccd8e17-qvkk5\" (UID: \"ca01721d-5c8f-4127-977f-5fa5eaf3100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccd8e17-qvkk5"
Nov 03 00:18:09.453589 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:09.453558    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccd8e17-qvkk5"
Nov 03 00:18:09.765648 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:09.765619    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ccd8e17-qvkk5"]
Nov 03 00:18:09.767320 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:18:09.767286    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca01721d_5c8f_4127_977f_5fa5eaf3100e.slice/crio-f4023f4c88772e383ececf5432cf3aba6641e4c2d8e925e878ced2ac41843924 WatchSource:0}: Error finding container f4023f4c88772e383ececf5432cf3aba6641e4c2d8e925e878ced2ac41843924: Status 404 returned error can't find the container with id f4023f4c88772e383ececf5432cf3aba6641e4c2d8e925e878ced2ac41843924
Nov 03 00:18:10.220902 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:10.220872    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccd8e17-qvkk5" event={"ID":"ca01721d-5c8f-4127-977f-5fa5eaf3100e","Type":"ContainerStarted","Data":"f701c34fee22053aacea1b673def012fd5c70199cd5b0aff016af302e9daba69"}
Nov 03 00:18:10.221260 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:10.220947    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccd8e17-qvkk5" event={"ID":"ca01721d-5c8f-4127-977f-5fa5eaf3100e","Type":"ContainerStarted","Data":"f4023f4c88772e383ececf5432cf3aba6641e4c2d8e925e878ced2ac41843924"}
Nov 03 00:18:10.256532 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:10.256480    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccd8e17-qvkk5" podStartSLOduration=1.2564639340000001 podStartE2EDuration="1.256463934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:10.254135091 +0000 UTC m=+773.133719135" watchObservedRunningTime="2026-11-03 00:18:10.256463934 +0000 UTC m=+773.136047976"
Nov 03 00:18:11.229097 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:11.229062    5012 generic.go:356] "Generic (PLEG): container finished" podID="ca01721d-5c8f-4127-977f-5fa5eaf3100e" containerID="f701c34fee22053aacea1b673def012fd5c70199cd5b0aff016af302e9daba69" exitCode=0
Nov 03 00:18:11.229491 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:11.229147    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccd8e17-qvkk5" event={"ID":"ca01721d-5c8f-4127-977f-5fa5eaf3100e","Type":"ContainerDied","Data":"f701c34fee22053aacea1b673def012fd5c70199cd5b0aff016af302e9daba69"}
Nov 03 00:18:12.382452 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:12.382429    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccd8e17-qvkk5"
Nov 03 00:18:12.514747 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:12.514669    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca01721d-5c8f-4127-977f-5fa5eaf3100e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca01721d-5c8f-4127-977f-5fa5eaf3100e-maas-api-xks-ca\") pod \"ca01721d-5c8f-4127-977f-5fa5eaf3100e\" (UID: \"ca01721d-5c8f-4127-977f-5fa5eaf3100e\") "
Nov 03 00:18:12.514747 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:12.514722    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca01721d-5c8f-4127-977f-5fa5eaf3100e-kube-api-access-ng6rt\" (UniqueName: \"kubernetes.io/projected/ca01721d-5c8f-4127-977f-5fa5eaf3100e-kube-api-access-ng6rt\") pod \"ca01721d-5c8f-4127-977f-5fa5eaf3100e\" (UID: \"ca01721d-5c8f-4127-977f-5fa5eaf3100e\") "
Nov 03 00:18:12.514955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:12.514769    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca01721d-5c8f-4127-977f-5fa5eaf3100e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca01721d-5c8f-4127-977f-5fa5eaf3100e-maas-api-openshift-ca\") pod \"ca01721d-5c8f-4127-977f-5fa5eaf3100e\" (UID: \"ca01721d-5c8f-4127-977f-5fa5eaf3100e\") "
Nov 03 00:18:12.515231 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:12.515207    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca01721d-5c8f-4127-977f-5fa5eaf3100e-maas-api-openshift-ca" pod "ca01721d-5c8f-4127-977f-5fa5eaf3100e" (UID: "ca01721d-5c8f-4127-977f-5fa5eaf3100e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:18:12.516687 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:12.516659    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca01721d-5c8f-4127-977f-5fa5eaf3100e-maas-api-xks-ca" pod "ca01721d-5c8f-4127-977f-5fa5eaf3100e" (UID: "ca01721d-5c8f-4127-977f-5fa5eaf3100e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:18:12.516792 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:12.516755    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca01721d-5c8f-4127-977f-5fa5eaf3100e-kube-api-access-ng6rt" pod "ca01721d-5c8f-4127-977f-5fa5eaf3100e" (UID: "ca01721d-5c8f-4127-977f-5fa5eaf3100e"). InnerVolumeSpecName "kube-api-access-ng6rt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:18:12.616207 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:12.616171    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng6rt\" (UniqueName: \"kubernetes.io/projected/ca01721d-5c8f-4127-977f-5fa5eaf3100e-kube-api-access-ng6rt\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:18:12.616207 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:12.616201    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca01721d-5c8f-4127-977f-5fa5eaf3100e-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:18:12.616376 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:12.616215    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca01721d-5c8f-4127-977f-5fa5eaf3100e-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:18:13.243520 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:13.243488    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccd8e17-qvkk5" event={"ID":"ca01721d-5c8f-4127-977f-5fa5eaf3100e","Type":"ContainerDied","Data":"f4023f4c88772e383ececf5432cf3aba6641e4c2d8e925e878ced2ac41843924"}
Nov 03 00:18:13.243685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:13.243540    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4023f4c88772e383ececf5432cf3aba6641e4c2d8e925e878ced2ac41843924"
Nov 03 00:18:13.243685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:13.243497    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ccd8e17-qvkk5"
Nov 03 00:18:14.848287 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:14.848250    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16172f30-lp8gm"]
Nov 03 00:18:14.848853 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:14.848838    5012 cpu_manager.go:414] "containerMap: removing container" podUID="ca01721d-5c8f-4127-977f-5fa5eaf3100e" containerName="revoke-keys"
Nov 03 00:18:14.848898 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:14.848855    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca01721d-5c8f-4127-977f-5fa5eaf3100e" containerName="revoke-keys"
Nov 03 00:18:14.848966 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:14.848956    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca01721d-5c8f-4127-977f-5fa5eaf3100e" containerName="revoke-keys"
Nov 03 00:18:14.852549 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:14.852532    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16172f30-lp8gm"
Nov 03 00:18:14.870902 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:14.870873    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16172f30-lp8gm"]
Nov 03 00:18:14.876949 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:14.876358    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 00:18:14.876949 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:14.876429    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:18:14.876949 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:14.876512    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 00:18:14.876949 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:14.876571    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:18:14.939762 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:14.939720    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad67a5bf-0df3-4d50-ad9d-52f2cc529566-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16172f30-lp8gm\" (UID: \"ad67a5bf-0df3-4d50-ad9d-52f2cc529566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16172f30-lp8gm"
Nov 03 00:18:14.939890 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:14.939785    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vpc8\" (UniqueName: \"kubernetes.io/projected/ad67a5bf-0df3-4d50-ad9d-52f2cc529566-kube-api-access-2vpc8\") pod \"maas-api-revoke-sub-16172f30-lp8gm\" (UID: \"ad67a5bf-0df3-4d50-ad9d-52f2cc529566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16172f30-lp8gm"
Nov 03 00:18:14.939963 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:14.939888    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad67a5bf-0df3-4d50-ad9d-52f2cc529566-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16172f30-lp8gm\" (UID: \"ad67a5bf-0df3-4d50-ad9d-52f2cc529566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16172f30-lp8gm"
Nov 03 00:18:15.040901 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:15.040871    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad67a5bf-0df3-4d50-ad9d-52f2cc529566-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16172f30-lp8gm\" (UID: \"ad67a5bf-0df3-4d50-ad9d-52f2cc529566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16172f30-lp8gm"
Nov 03 00:18:15.041080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:15.040959    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vpc8\" (UniqueName: \"kubernetes.io/projected/ad67a5bf-0df3-4d50-ad9d-52f2cc529566-kube-api-access-2vpc8\") pod \"maas-api-revoke-sub-16172f30-lp8gm\" (UID: \"ad67a5bf-0df3-4d50-ad9d-52f2cc529566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16172f30-lp8gm"
Nov 03 00:18:15.041080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:15.040999    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad67a5bf-0df3-4d50-ad9d-52f2cc529566-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16172f30-lp8gm\" (UID: \"ad67a5bf-0df3-4d50-ad9d-52f2cc529566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16172f30-lp8gm"
Nov 03 00:18:15.041601 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:15.041573    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad67a5bf-0df3-4d50-ad9d-52f2cc529566-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16172f30-lp8gm\" (UID: \"ad67a5bf-0df3-4d50-ad9d-52f2cc529566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16172f30-lp8gm"
Nov 03 00:18:15.043222 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:15.043199    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad67a5bf-0df3-4d50-ad9d-52f2cc529566-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16172f30-lp8gm\" (UID: \"ad67a5bf-0df3-4d50-ad9d-52f2cc529566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16172f30-lp8gm"
Nov 03 00:18:15.058542 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:15.058513    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vpc8\" (UniqueName: \"kubernetes.io/projected/ad67a5bf-0df3-4d50-ad9d-52f2cc529566-kube-api-access-2vpc8\") pod \"maas-api-revoke-sub-16172f30-lp8gm\" (UID: \"ad67a5bf-0df3-4d50-ad9d-52f2cc529566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16172f30-lp8gm"
Nov 03 00:18:15.166491 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:15.166420    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16172f30-lp8gm"
Nov 03 00:18:15.462833 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:15.462805    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16172f30-lp8gm"]
Nov 03 00:18:15.463942 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:18:15.463899    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad67a5bf_0df3_4d50_ad9d_52f2cc529566.slice/crio-c3a4f2cd4fa21ac99e4c354799fe79c2bdaa5d024168e57b4b02f0da35a3c1d2 WatchSource:0}: Error finding container c3a4f2cd4fa21ac99e4c354799fe79c2bdaa5d024168e57b4b02f0da35a3c1d2: Status 404 returned error can't find the container with id c3a4f2cd4fa21ac99e4c354799fe79c2bdaa5d024168e57b4b02f0da35a3c1d2
Nov 03 00:18:16.267148 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:16.267109    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16172f30-lp8gm" event={"ID":"ad67a5bf-0df3-4d50-ad9d-52f2cc529566","Type":"ContainerStarted","Data":"03b138aca35431f3563693f27ad115d78141f8d271e244f5040e462bd15670eb"}
Nov 03 00:18:16.267539 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:16.267165    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16172f30-lp8gm" event={"ID":"ad67a5bf-0df3-4d50-ad9d-52f2cc529566","Type":"ContainerStarted","Data":"c3a4f2cd4fa21ac99e4c354799fe79c2bdaa5d024168e57b4b02f0da35a3c1d2"}
Nov 03 00:18:16.300981 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:16.300720    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16172f30-lp8gm" podStartSLOduration=2.300706908 podStartE2EDuration="2.300706908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:16.298698431 +0000 UTC m=+779.178282531" watchObservedRunningTime="2026-11-03 00:18:16.300706908 +0000 UTC m=+779.180290950"
Nov 03 00:18:17.281626 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:17.281590    5012 generic.go:356] "Generic (PLEG): container finished" podID="ad67a5bf-0df3-4d50-ad9d-52f2cc529566" containerID="03b138aca35431f3563693f27ad115d78141f8d271e244f5040e462bd15670eb" exitCode=0
Nov 03 00:18:17.282029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:17.281671    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16172f30-lp8gm" event={"ID":"ad67a5bf-0df3-4d50-ad9d-52f2cc529566","Type":"ContainerDied","Data":"03b138aca35431f3563693f27ad115d78141f8d271e244f5040e462bd15670eb"}
Nov 03 00:18:18.442327 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:18.442305    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16172f30-lp8gm"
Nov 03 00:18:18.511416 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:18.511377    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad67a5bf-0df3-4d50-ad9d-52f2cc529566-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad67a5bf-0df3-4d50-ad9d-52f2cc529566-maas-api-xks-ca\") pod \"ad67a5bf-0df3-4d50-ad9d-52f2cc529566\" (UID: \"ad67a5bf-0df3-4d50-ad9d-52f2cc529566\") "
Nov 03 00:18:18.511558 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:18.511432    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad67a5bf-0df3-4d50-ad9d-52f2cc529566-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad67a5bf-0df3-4d50-ad9d-52f2cc529566-maas-api-openshift-ca\") pod \"ad67a5bf-0df3-4d50-ad9d-52f2cc529566\" (UID: \"ad67a5bf-0df3-4d50-ad9d-52f2cc529566\") "
Nov 03 00:18:18.511628 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:18.511572    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad67a5bf-0df3-4d50-ad9d-52f2cc529566-kube-api-access-2vpc8\" (UniqueName: \"kubernetes.io/projected/ad67a5bf-0df3-4d50-ad9d-52f2cc529566-kube-api-access-2vpc8\") pod \"ad67a5bf-0df3-4d50-ad9d-52f2cc529566\" (UID: \"ad67a5bf-0df3-4d50-ad9d-52f2cc529566\") "
Nov 03 00:18:18.511918 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:18.511857    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad67a5bf-0df3-4d50-ad9d-52f2cc529566-maas-api-openshift-ca" pod "ad67a5bf-0df3-4d50-ad9d-52f2cc529566" (UID: "ad67a5bf-0df3-4d50-ad9d-52f2cc529566"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:18:18.512087 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:18.512016    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad67a5bf-0df3-4d50-ad9d-52f2cc529566-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:18:18.513898 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:18.513877    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad67a5bf-0df3-4d50-ad9d-52f2cc529566-maas-api-xks-ca" pod "ad67a5bf-0df3-4d50-ad9d-52f2cc529566" (UID: "ad67a5bf-0df3-4d50-ad9d-52f2cc529566"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:18:18.514176 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:18.514152    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad67a5bf-0df3-4d50-ad9d-52f2cc529566-kube-api-access-2vpc8" pod "ad67a5bf-0df3-4d50-ad9d-52f2cc529566" (UID: "ad67a5bf-0df3-4d50-ad9d-52f2cc529566"). InnerVolumeSpecName "kube-api-access-2vpc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:18:18.613000 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:18.612972    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad67a5bf-0df3-4d50-ad9d-52f2cc529566-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:18:18.613000 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:18.612997    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vpc8\" (UniqueName: \"kubernetes.io/projected/ad67a5bf-0df3-4d50-ad9d-52f2cc529566-kube-api-access-2vpc8\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:18:19.299760 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.299733    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16172f30-lp8gm"
Nov 03 00:18:19.299889 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.299768    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16172f30-lp8gm" event={"ID":"ad67a5bf-0df3-4d50-ad9d-52f2cc529566","Type":"ContainerDied","Data":"c3a4f2cd4fa21ac99e4c354799fe79c2bdaa5d024168e57b4b02f0da35a3c1d2"}
Nov 03 00:18:19.299889 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.299808    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3a4f2cd4fa21ac99e4c354799fe79c2bdaa5d024168e57b4b02f0da35a3c1d2"
Nov 03 00:18:19.343476 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.343449    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c"]
Nov 03 00:18:19.343945 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.343929    5012 cpu_manager.go:414] "containerMap: removing container" podUID="ad67a5bf-0df3-4d50-ad9d-52f2cc529566" containerName="revoke-keys"
Nov 03 00:18:19.343945 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.343943    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad67a5bf-0df3-4d50-ad9d-52f2cc529566" containerName="revoke-keys"
Nov 03 00:18:19.344050 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.344041    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad67a5bf-0df3-4d50-ad9d-52f2cc529566" containerName="revoke-keys"
Nov 03 00:18:19.347383 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.347366    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c"
Nov 03 00:18:19.371593 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.371531    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b39c24993f8f\""
Nov 03 00:18:19.371748 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.371603    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9bacec7b4880\""
Nov 03 00:18:19.374711 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.374690    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-xtc2k\""
Nov 03 00:18:19.381488 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.381397    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c"]
Nov 03 00:18:19.408356 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.407375    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 00:18:19.408356 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.407495    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 00:18:19.408356 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.407557    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 00:18:19.408356 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.407713    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 00:18:19.420843 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.420812    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8phsq\" (UniqueName: \"kubernetes.io/projected/ec632772-d93a-4cfe-a905-433fc4ed7024-kube-api-access-8phsq\") pod \"payload-processing-external-model-cdc4d9fcc-xrv2c\" (UID: \"ec632772-d93a-4cfe-a905-433fc4ed7024\") " pod="models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c"
Nov 03 00:18:19.421086 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.421063    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec632772-d93a-4cfe-a905-433fc4ed7024-routing-overlay-volume\") pod \"payload-processing-external-model-cdc4d9fcc-xrv2c\" (UID: \"ec632772-d93a-4cfe-a905-433fc4ed7024\") " pod="models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c"
Nov 03 00:18:19.421256 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.421238    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec632772-d93a-4cfe-a905-433fc4ed7024-provider-credentials\") pod \"payload-processing-external-model-cdc4d9fcc-xrv2c\" (UID: \"ec632772-d93a-4cfe-a905-433fc4ed7024\") " pod="models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c"
Nov 03 00:18:19.421432 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.421414    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec632772-d93a-4cfe-a905-433fc4ed7024-plugins-config-volume\") pod \"payload-processing-external-model-cdc4d9fcc-xrv2c\" (UID: \"ec632772-d93a-4cfe-a905-433fc4ed7024\") " pod="models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c"
Nov 03 00:18:19.522308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.522220    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec632772-d93a-4cfe-a905-433fc4ed7024-plugins-config-volume\") pod \"payload-processing-external-model-cdc4d9fcc-xrv2c\" (UID: \"ec632772-d93a-4cfe-a905-433fc4ed7024\") " pod="models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c"
Nov 03 00:18:19.522712 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.522333    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8phsq\" (UniqueName: \"kubernetes.io/projected/ec632772-d93a-4cfe-a905-433fc4ed7024-kube-api-access-8phsq\") pod \"payload-processing-external-model-cdc4d9fcc-xrv2c\" (UID: \"ec632772-d93a-4cfe-a905-433fc4ed7024\") " pod="models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c"
Nov 03 00:18:19.522712 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.522378    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec632772-d93a-4cfe-a905-433fc4ed7024-routing-overlay-volume\") pod \"payload-processing-external-model-cdc4d9fcc-xrv2c\" (UID: \"ec632772-d93a-4cfe-a905-433fc4ed7024\") " pod="models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c"
Nov 03 00:18:19.522712 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.522426    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec632772-d93a-4cfe-a905-433fc4ed7024-provider-credentials\") pod \"payload-processing-external-model-cdc4d9fcc-xrv2c\" (UID: \"ec632772-d93a-4cfe-a905-433fc4ed7024\") " pod="models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c"
Nov 03 00:18:19.523096 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.523068    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec632772-d93a-4cfe-a905-433fc4ed7024-routing-overlay-volume\") pod \"payload-processing-external-model-cdc4d9fcc-xrv2c\" (UID: \"ec632772-d93a-4cfe-a905-433fc4ed7024\") " pod="models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c"
Nov 03 00:18:19.523096 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.523087    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec632772-d93a-4cfe-a905-433fc4ed7024-plugins-config-volume\") pod \"payload-processing-external-model-cdc4d9fcc-xrv2c\" (UID: \"ec632772-d93a-4cfe-a905-433fc4ed7024\") " pod="models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c"
Nov 03 00:18:19.524879 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.524853    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec632772-d93a-4cfe-a905-433fc4ed7024-provider-credentials\") pod \"payload-processing-external-model-cdc4d9fcc-xrv2c\" (UID: \"ec632772-d93a-4cfe-a905-433fc4ed7024\") " pod="models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c"
Nov 03 00:18:19.539360 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.539341    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8phsq\" (UniqueName: \"kubernetes.io/projected/ec632772-d93a-4cfe-a905-433fc4ed7024-kube-api-access-8phsq\") pod \"payload-processing-external-model-cdc4d9fcc-xrv2c\" (UID: \"ec632772-d93a-4cfe-a905-433fc4ed7024\") " pod="models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c"
Nov 03 00:18:19.660822 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.660783    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c"
Nov 03 00:18:19.960365 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:19.960340    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c"]
Nov 03 00:18:19.962199 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:18:19.962176    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec632772_d93a_4cfe_a905_433fc4ed7024.slice/crio-3f23bc3b8b4396668e00d97ed44ec5684cad632257c1349430cbfe780421a0ab WatchSource:0}: Error finding container 3f23bc3b8b4396668e00d97ed44ec5684cad632257c1349430cbfe780421a0ab: Status 404 returned error can't find the container with id 3f23bc3b8b4396668e00d97ed44ec5684cad632257c1349430cbfe780421a0ab
Nov 03 00:18:20.308351 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:20.308252    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c" event={"ID":"ec632772-d93a-4cfe-a905-433fc4ed7024","Type":"ContainerStarted","Data":"3dfe60f793b603dfced9adcfc48015f271f95604c2529190eb4a9c700e8b6bbc"}
Nov 03 00:18:20.308351 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:20.308348    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c" event={"ID":"ec632772-d93a-4cfe-a905-433fc4ed7024","Type":"ContainerStarted","Data":"3f23bc3b8b4396668e00d97ed44ec5684cad632257c1349430cbfe780421a0ab"}
Nov 03 00:18:20.308522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:20.308455    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c"
Nov 03 00:18:20.344435 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:20.344390    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c" podStartSLOduration=1.344377808 podStartE2EDuration="1.344377808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:20.340487966 +0000 UTC m=+783.220072008" watchObservedRunningTime="2026-11-03 00:18:20.344377808 +0000 UTC m=+783.223961867"
Nov 03 00:18:23.453570 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:23.453537    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6c77ced-b5sfz"]
Nov 03 00:18:23.457616 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:23.457600    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c77ced-b5sfz"
Nov 03 00:18:23.469809 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:23.469780    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:18:23.473890 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:23.473867    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6c77ced-b5sfz"]
Nov 03 00:18:23.480061 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:23.479751    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:18:23.480061 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:23.479802    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 00:18:23.480421 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:23.480399    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 00:18:23.565012 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:23.564986    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3251a5de-51bf-4066-804f-f028a0109834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6c77ced-b5sfz\" (UID: \"3251a5de-51bf-4066-804f-f028a0109834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c77ced-b5sfz"
Nov 03 00:18:23.565104 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:23.565017    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhwl5\" (UniqueName: \"kubernetes.io/projected/3251a5de-51bf-4066-804f-f028a0109834-kube-api-access-dhwl5\") pod \"maas-api-revoke-sub-d6c77ced-b5sfz\" (UID: \"3251a5de-51bf-4066-804f-f028a0109834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c77ced-b5sfz"
Nov 03 00:18:23.565104 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:23.565049    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3251a5de-51bf-4066-804f-f028a0109834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6c77ced-b5sfz\" (UID: \"3251a5de-51bf-4066-804f-f028a0109834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c77ced-b5sfz"
Nov 03 00:18:23.666371 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:23.666338    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3251a5de-51bf-4066-804f-f028a0109834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6c77ced-b5sfz\" (UID: \"3251a5de-51bf-4066-804f-f028a0109834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c77ced-b5sfz"
Nov 03 00:18:23.666539 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:23.666375    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhwl5\" (UniqueName: \"kubernetes.io/projected/3251a5de-51bf-4066-804f-f028a0109834-kube-api-access-dhwl5\") pod \"maas-api-revoke-sub-d6c77ced-b5sfz\" (UID: \"3251a5de-51bf-4066-804f-f028a0109834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c77ced-b5sfz"
Nov 03 00:18:23.666539 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:23.666495    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3251a5de-51bf-4066-804f-f028a0109834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6c77ced-b5sfz\" (UID: \"3251a5de-51bf-4066-804f-f028a0109834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c77ced-b5sfz"
Nov 03 00:18:23.667021 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:23.667002    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3251a5de-51bf-4066-804f-f028a0109834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6c77ced-b5sfz\" (UID: \"3251a5de-51bf-4066-804f-f028a0109834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c77ced-b5sfz"
Nov 03 00:18:23.668832 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:23.668807    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3251a5de-51bf-4066-804f-f028a0109834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6c77ced-b5sfz\" (UID: \"3251a5de-51bf-4066-804f-f028a0109834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c77ced-b5sfz"
Nov 03 00:18:23.682888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:23.682863    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhwl5\" (UniqueName: \"kubernetes.io/projected/3251a5de-51bf-4066-804f-f028a0109834-kube-api-access-dhwl5\") pod \"maas-api-revoke-sub-d6c77ced-b5sfz\" (UID: \"3251a5de-51bf-4066-804f-f028a0109834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c77ced-b5sfz"
Nov 03 00:18:23.771305 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:23.771252    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c77ced-b5sfz"
Nov 03 00:18:24.067221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:24.067189    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6c77ced-b5sfz"]
Nov 03 00:18:24.067609 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:18:24.067587    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3251a5de_51bf_4066_804f_f028a0109834.slice/crio-6344e7b77c2cc9bb6fc73300a74a9406d7c916d10fc055dc72ec9025ab3e2e23 WatchSource:0}: Error finding container 6344e7b77c2cc9bb6fc73300a74a9406d7c916d10fc055dc72ec9025ab3e2e23: Status 404 returned error can't find the container with id 6344e7b77c2cc9bb6fc73300a74a9406d7c916d10fc055dc72ec9025ab3e2e23
Nov 03 00:18:24.335498 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:24.335412    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c77ced-b5sfz" event={"ID":"3251a5de-51bf-4066-804f-f028a0109834","Type":"ContainerStarted","Data":"6344e7b77c2cc9bb6fc73300a74a9406d7c916d10fc055dc72ec9025ab3e2e23"}
Nov 03 00:18:25.347865 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:25.347831    5012 generic.go:356] "Generic (PLEG): container finished" podID="3251a5de-51bf-4066-804f-f028a0109834" containerID="1685539c0d50e74c3743c0bd821a8972ac56291f468197ca852ef6755b98bd75" exitCode=0
Nov 03 00:18:25.348378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:25.347902    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c77ced-b5sfz" event={"ID":"3251a5de-51bf-4066-804f-f028a0109834","Type":"ContainerDied","Data":"1685539c0d50e74c3743c0bd821a8972ac56291f468197ca852ef6755b98bd75"}
Nov 03 00:18:26.557954 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:26.557924    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c77ced-b5sfz"
Nov 03 00:18:26.607990 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:26.607953    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3251a5de-51bf-4066-804f-f028a0109834-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3251a5de-51bf-4066-804f-f028a0109834-maas-api-xks-ca\") pod \"3251a5de-51bf-4066-804f-f028a0109834\" (UID: \"3251a5de-51bf-4066-804f-f028a0109834\") "
Nov 03 00:18:26.608159 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:26.608143    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3251a5de-51bf-4066-804f-f028a0109834-kube-api-access-dhwl5\" (UniqueName: \"kubernetes.io/projected/3251a5de-51bf-4066-804f-f028a0109834-kube-api-access-dhwl5\") pod \"3251a5de-51bf-4066-804f-f028a0109834\" (UID: \"3251a5de-51bf-4066-804f-f028a0109834\") "
Nov 03 00:18:26.608241 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:26.608220    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3251a5de-51bf-4066-804f-f028a0109834-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3251a5de-51bf-4066-804f-f028a0109834-maas-api-openshift-ca\") pod \"3251a5de-51bf-4066-804f-f028a0109834\" (UID: \"3251a5de-51bf-4066-804f-f028a0109834\") "
Nov 03 00:18:26.609009 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:26.608966    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3251a5de-51bf-4066-804f-f028a0109834-maas-api-openshift-ca" pod "3251a5de-51bf-4066-804f-f028a0109834" (UID: "3251a5de-51bf-4066-804f-f028a0109834"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:18:26.611336 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:26.611258    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3251a5de-51bf-4066-804f-f028a0109834-maas-api-xks-ca" pod "3251a5de-51bf-4066-804f-f028a0109834" (UID: "3251a5de-51bf-4066-804f-f028a0109834"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:18:26.611749 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:26.611723    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3251a5de-51bf-4066-804f-f028a0109834-kube-api-access-dhwl5" pod "3251a5de-51bf-4066-804f-f028a0109834" (UID: "3251a5de-51bf-4066-804f-f028a0109834"). InnerVolumeSpecName "kube-api-access-dhwl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:18:26.709819 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:26.709790    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhwl5\" (UniqueName: \"kubernetes.io/projected/3251a5de-51bf-4066-804f-f028a0109834-kube-api-access-dhwl5\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:18:26.709819 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:26.709815    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3251a5de-51bf-4066-804f-f028a0109834-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:18:26.709819 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:26.709825    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3251a5de-51bf-4066-804f-f028a0109834-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:18:27.368101 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:27.368071    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c77ced-b5sfz"
Nov 03 00:18:27.368283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:27.368105    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6c77ced-b5sfz" event={"ID":"3251a5de-51bf-4066-804f-f028a0109834","Type":"ContainerDied","Data":"6344e7b77c2cc9bb6fc73300a74a9406d7c916d10fc055dc72ec9025ab3e2e23"}
Nov 03 00:18:27.368283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:27.368158    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6344e7b77c2cc9bb6fc73300a74a9406d7c916d10fc055dc72ec9025ab3e2e23"
Nov 03 00:18:31.319532 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:31.319501    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c"
Nov 03 00:18:37.290371 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.290323    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"]
Nov 03 00:18:37.291815 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.291789    5012 cpu_manager.go:414] "containerMap: removing container" podUID="3251a5de-51bf-4066-804f-f028a0109834" containerName="revoke-keys"
Nov 03 00:18:37.291969 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.291953    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3251a5de-51bf-4066-804f-f028a0109834" containerName="revoke-keys"
Nov 03 00:18:37.292253 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.292236    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="3251a5de-51bf-4066-804f-f028a0109834" containerName="revoke-keys"
Nov 03 00:18:37.297322 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.297301    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:18:37.315413 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.315382    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"]
Nov 03 00:18:37.321677 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.321027    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8e90c5\"/\"e2e-conflict-a-5f6115a4-kserve-self-signed-certs\""
Nov 03 00:18:37.391293 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:18:37.391236    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:18:37.430274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.430197    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-model-cache\") pod \"e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq\" (UID: \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:18:37.430597 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.430572    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-tmp-dir\") pod \"e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq\" (UID: \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:18:37.430770 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.430742    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d6rk\" (UniqueName: \"kubernetes.io/projected/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-kube-api-access-9d6rk\") pod \"e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq\" (UID: \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:18:37.430993 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.430896    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-home\") pod \"e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq\" (UID: \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:18:37.431168 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.431150    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-dshm\") pod \"e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq\" (UID: \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:18:37.431326 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.431308    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-tls-certs\") pod \"e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq\" (UID: \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:18:37.532540 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.532499    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d6rk\" (UniqueName: \"kubernetes.io/projected/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-kube-api-access-9d6rk\") pod \"e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq\" (UID: \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:18:37.532854 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.532828    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-home\") pod \"e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq\" (UID: \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:18:37.533047 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.533027    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-dshm\") pod \"e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq\" (UID: \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:18:37.533216 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.533198    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-tls-certs\") pod \"e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq\" (UID: \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:18:37.533378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.533361    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-model-cache\") pod \"e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq\" (UID: \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:18:37.533559 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.533543    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-tmp-dir\") pod \"e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq\" (UID: \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:18:37.534129 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.534106    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-tmp-dir\") pod \"e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq\" (UID: \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:18:37.534827 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.534805    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-home\") pod \"e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq\" (UID: \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:18:37.535871 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.535850    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-model-cache\") pod \"e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq\" (UID: \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:18:37.560011 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.558208    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-dshm\") pod \"e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq\" (UID: \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:18:37.560011 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.559019    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-tls-certs\") pod \"e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq\" (UID: \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:18:37.561981 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.561956    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d6rk\" (UniqueName: \"kubernetes.io/projected/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-kube-api-access-9d6rk\") pod \"e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq\" (UID: \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:18:37.625899 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.625870    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:18:37.954320 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:37.954292    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"]
Nov 03 00:18:37.956091 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:18:37.956064    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb53f3e6b_ee4f_4fbc_8d8e_09b783764c4b.slice/crio-f07ad87faa14975348c9ffa1a3f26c2646d6e211f3d6dec4803b3e8158994334 WatchSource:0}: Error finding container f07ad87faa14975348c9ffa1a3f26c2646d6e211f3d6dec4803b3e8158994334: Status 404 returned error can't find the container with id f07ad87faa14975348c9ffa1a3f26c2646d6e211f3d6dec4803b3e8158994334
Nov 03 00:18:38.453621 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:38.453586    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq" event={"ID":"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b","Type":"ContainerStarted","Data":"66f5bbca51de8c2d3a4492897ce8539e5a9d897720a2d7f68b732b2acf3a320f"}
Nov 03 00:18:38.454054 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:38.453630    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq" event={"ID":"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b","Type":"ContainerStarted","Data":"f07ad87faa14975348c9ffa1a3f26c2646d6e211f3d6dec4803b3e8158994334"}
Nov 03 00:18:38.508464 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:38.508415    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq" podStartSLOduration=1.508402434 podStartE2EDuration="1.508402434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:38.507463508 +0000 UTC m=+801.387047566" watchObservedRunningTime="2026-11-03 00:18:38.508402434 +0000 UTC m=+801.387986476"
Nov 03 00:18:38.626775 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:38.626750    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:18:39.397680 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:39.397646    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx"]
Nov 03 00:18:39.401514 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:39.401493    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx"
Nov 03 00:18:39.418995 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:39.418961    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx"]
Nov 03 00:18:39.559100 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:39.559069    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-plugins-config-volume\") pod \"payload-processing-external-model-865949b6c4-vjfgx\" (UID: \"0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55\") " pod="models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx"
Nov 03 00:18:39.559468 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:39.559147    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-routing-overlay-volume\") pod \"payload-processing-external-model-865949b6c4-vjfgx\" (UID: \"0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55\") " pod="models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx"
Nov 03 00:18:39.559468 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:39.559352    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j55v\" (UniqueName: \"kubernetes.io/projected/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-kube-api-access-9j55v\") pod \"payload-processing-external-model-865949b6c4-vjfgx\" (UID: \"0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55\") " pod="models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx"
Nov 03 00:18:39.559468 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:39.559402    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-provider-credentials\") pod \"payload-processing-external-model-865949b6c4-vjfgx\" (UID: \"0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55\") " pod="models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx"
Nov 03 00:18:39.660293 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:39.660230    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-plugins-config-volume\") pod \"payload-processing-external-model-865949b6c4-vjfgx\" (UID: \"0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55\") " pod="models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx"
Nov 03 00:18:39.660293 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:39.660275    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-routing-overlay-volume\") pod \"payload-processing-external-model-865949b6c4-vjfgx\" (UID: \"0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55\") " pod="models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx"
Nov 03 00:18:39.660481 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:39.660303    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j55v\" (UniqueName: \"kubernetes.io/projected/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-kube-api-access-9j55v\") pod \"payload-processing-external-model-865949b6c4-vjfgx\" (UID: \"0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55\") " pod="models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx"
Nov 03 00:18:39.660481 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:39.660329    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-provider-credentials\") pod \"payload-processing-external-model-865949b6c4-vjfgx\" (UID: \"0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55\") " pod="models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx"
Nov 03 00:18:39.661155 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:39.661123    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-routing-overlay-volume\") pod \"payload-processing-external-model-865949b6c4-vjfgx\" (UID: \"0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55\") " pod="models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx"
Nov 03 00:18:39.661274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:39.661181    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-plugins-config-volume\") pod \"payload-processing-external-model-865949b6c4-vjfgx\" (UID: \"0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55\") " pod="models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx"
Nov 03 00:18:39.662887 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:39.662853    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-provider-credentials\") pod \"payload-processing-external-model-865949b6c4-vjfgx\" (UID: \"0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55\") " pod="models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx"
Nov 03 00:18:39.677151 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:39.677125    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j55v\" (UniqueName: \"kubernetes.io/projected/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-kube-api-access-9j55v\") pod \"payload-processing-external-model-865949b6c4-vjfgx\" (UID: \"0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55\") " pod="models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx"
Nov 03 00:18:39.715242 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:39.715218    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx"
Nov 03 00:18:40.010627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:40.010601    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx"]
Nov 03 00:18:40.011534 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:18:40.011513    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c579e4b_ed0b_4ed2_83f9_e644e7f1ee55.slice/crio-1aa80c3376c3e8f63362994e33c072dbbc99b84a70b166fd8c64b2395159a5f2 WatchSource:0}: Error finding container 1aa80c3376c3e8f63362994e33c072dbbc99b84a70b166fd8c64b2395159a5f2: Status 404 returned error can't find the container with id 1aa80c3376c3e8f63362994e33c072dbbc99b84a70b166fd8c64b2395159a5f2
Nov 03 00:18:40.470113 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:40.470062    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx" event={"ID":"0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55","Type":"ContainerStarted","Data":"3a69352c31554ff30543f00ef7f7e75fcc8dadfbeb977cf2c939cdada83bc4c6"}
Nov 03 00:18:40.470296 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:40.470128    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx" event={"ID":"0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55","Type":"ContainerStarted","Data":"1aa80c3376c3e8f63362994e33c072dbbc99b84a70b166fd8c64b2395159a5f2"}
Nov 03 00:18:40.470296 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:40.470155    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx"
Nov 03 00:18:40.504628 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:40.504579    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx" podStartSLOduration=1.504565435 podStartE2EDuration="1.504565435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:40.503292951 +0000 UTC m=+803.382877004" watchObservedRunningTime="2026-11-03 00:18:40.504565435 +0000 UTC m=+803.384149477"
Nov 03 00:18:47.626074 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.626040    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:18:47.638763 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.638737    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:18:47.883491 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.883405    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"]
Nov 03 00:18:47.888661 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.888639    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"]
Nov 03 00:18:47.888788 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.888761    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:47.892490 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.892466    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:47.911213 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.911178    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-4c61c8-openshift-default-dockercfg-sclgr\""
Nov 03 00:18:47.913124 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.913101    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"]
Nov 03 00:18:47.921042 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.921022    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"]
Nov 03 00:18:47.932255 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.932233    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"]
Nov 03 00:18:47.940400 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.940366    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:47.960435 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.960414    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"]
Nov 03 00:18:47.987255 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.987226    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:47.987361 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.987256    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:47.987361 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.987277    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:47.987361 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.987345    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:47.987473 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.987439    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:47.987510 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.987487    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:47.987544 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.987511    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:47.987577 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.987540    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:47.987610 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.987589    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:47.987645 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.987615    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:47.987675 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.987649    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:47.987708 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.987682    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:47.987757 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.987742    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:47.987805 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.987791    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:47.987850 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.987831    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6cg9\" (UniqueName: \"kubernetes.io/projected/876fffce-7b9b-4aeb-8437-4c60987a5056-kube-api-access-q6cg9\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:47.987888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.987863    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:47.987934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.987886    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m57hn\" (UniqueName: \"kubernetes.io/projected/19edcff3-65d5-489c-bb33-7ad5e4bcc791-kube-api-access-m57hn\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:47.987967 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.987928    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:47.987967 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.987958    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:47.988034 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.987986    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:47.988034 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.988011    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/876fffce-7b9b-4aeb-8437-4c60987a5056-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:47.988097 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.988036    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:47.988097 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.988071    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:47.988162 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.988115    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:47.988162 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.988147    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:47.988224 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.988170    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fn6h\" (UniqueName: \"kubernetes.io/projected/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-kube-api-access-2fn6h\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:47.988259 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:47.988224    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:48.089617 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.089592    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:48.089735 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.089630    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:48.089735 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.089653    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6cg9\" (UniqueName: \"kubernetes.io/projected/876fffce-7b9b-4aeb-8437-4c60987a5056-kube-api-access-q6cg9\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:48.089735 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.089672    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:48.089898 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.089780    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m57hn\" (UniqueName: \"kubernetes.io/projected/19edcff3-65d5-489c-bb33-7ad5e4bcc791-kube-api-access-m57hn\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:48.089898 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.089821    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:48.089898 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.089842    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:48.089898 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.089871    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:48.089898 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.089896    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/876fffce-7b9b-4aeb-8437-4c60987a5056-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:48.090202 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.089958    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:48.090202 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.089994    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:48.090202 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.090035    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:48.090202 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.090064    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:48.090202 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.090084    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:48.090202 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.090096    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fn6h\" (UniqueName: \"kubernetes.io/projected/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-kube-api-access-2fn6h\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:48.090202 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.090143    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:48.090557 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.090286    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:48.090557 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.090336    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:48.090557 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.090389    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:48.090557 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.090508    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:48.090749 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.090569    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:48.090749 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.090684    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:48.090749 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.090728    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:48.090749 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.090690    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:48.091193 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.091022    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:48.091348 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.091323    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:48.091469 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.091326    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/876fffce-7b9b-4aeb-8437-4c60987a5056-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:48.091664 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.091642    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:48.091722 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.091648    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:48.091778 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.091737    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:48.091830 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.091772    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:48.091830 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.091805    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:48.091949 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.091843    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:48.091949 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.091936    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:48.092050 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.091972    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:48.092050 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.092010    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:48.092154 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.092055    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:48.093411 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.093010    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:48.093411 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.093227    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:48.093572 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.093550    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:48.093813 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.093791    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:48.094076 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.094057    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:48.096197 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.094830    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:48.096197 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.095604    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:48.097952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.096736    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:48.097952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.097091    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:48.098183 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.098151    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:48.102855 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.102833    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:48.106737 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.106710    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:48.108073 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.108051    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:48.108336 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.108313    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fn6h\" (UniqueName: \"kubernetes.io/projected/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-kube-api-access-2fn6h\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:48.108401 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.108330    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-7fw2m\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:48.108401 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.108393    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6cg9\" (UniqueName: \"kubernetes.io/projected/876fffce-7b9b-4aeb-8437-4c60987a5056-kube-api-access-q6cg9\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:48.108515 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.108416    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m57hn\" (UniqueName: \"kubernetes.io/projected/19edcff3-65d5-489c-bb33-7ad5e4bcc791-kube-api-access-m57hn\") pod \"maas-default-gateway-openshift-default-6cc458d887-rgwqf\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:48.203628 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.203580    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:48.210657 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.210640    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:48.254118 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.254084    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:48.537192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.537120    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"]
Nov 03 00:18:48.541267 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:18:48.541243    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod876fffce_7b9b_4aeb_8437_4c60987a5056.slice/crio-35dd00a1792753bdb31c52c5c9c32c89e1ef10eedb20d92e6755cbe041913401 WatchSource:0}: Error finding container 35dd00a1792753bdb31c52c5c9c32c89e1ef10eedb20d92e6755cbe041913401: Status 404 returned error can't find the container with id 35dd00a1792753bdb31c52c5c9c32c89e1ef10eedb20d92e6755cbe041913401
Nov 03 00:18:48.543615 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.543582    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:18:48.543724 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.543660    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:18:48.543724 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.543705    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:18:48.546744 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.546721    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"]
Nov 03 00:18:48.549110 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:18:48.549083    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188d122e_1edb_44d5_8c9c_0119c8bb4e3b.slice/crio-be5f1c41594022b60ba7e8a4e2cdd7e8221f1944dfa24c50e9ae76d96a0d02d2 WatchSource:0}: Error finding container be5f1c41594022b60ba7e8a4e2cdd7e8221f1944dfa24c50e9ae76d96a0d02d2: Status 404 returned error can't find the container with id be5f1c41594022b60ba7e8a4e2cdd7e8221f1944dfa24c50e9ae76d96a0d02d2
Nov 03 00:18:48.552467 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.552444    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:18:48.554508 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.554476    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:18:48.554588 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.554557    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:18:48.554630 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.554601    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:18:48.597717 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.597694    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"]
Nov 03 00:18:48.600840 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.600818    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:18:48.600967 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.600891    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:18:48.601032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:48.600993    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:18:49.584736 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:49.584696    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf" event={"ID":"19edcff3-65d5-489c-bb33-7ad5e4bcc791","Type":"ContainerStarted","Data":"d1f9a1cce5798b8c619e50aca38ae8e03916658b119fbb673716f828abc6ff06"}
Nov 03 00:18:49.585249 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:49.584943    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf" event={"ID":"19edcff3-65d5-489c-bb33-7ad5e4bcc791","Type":"ContainerStarted","Data":"f4e2be576e01737fad6045b38af12a0f61c27c0e0f3be6cdd87d89f608af5275"}
Nov 03 00:18:49.603338 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:49.603303    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m" event={"ID":"188d122e-1edb-44d5-8c9c-0119c8bb4e3b","Type":"ContainerStarted","Data":"3c7905aa0cdfca4f747d30b5e5ef20170c69f5ed732a52bf797e07ed1c41bd46"}
Nov 03 00:18:49.603660 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:49.603634    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m" event={"ID":"188d122e-1edb-44d5-8c9c-0119c8bb4e3b","Type":"ContainerStarted","Data":"be5f1c41594022b60ba7e8a4e2cdd7e8221f1944dfa24c50e9ae76d96a0d02d2"}
Nov 03 00:18:49.621737 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:49.621702    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf" event={"ID":"876fffce-7b9b-4aeb-8437-4c60987a5056","Type":"ContainerStarted","Data":"42799281f276d4500ea0f6601e1e08f342f3eef49f4a9c1bf004253fe30c9f09"}
Nov 03 00:18:49.622005 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:49.621977    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf" event={"ID":"876fffce-7b9b-4aeb-8437-4c60987a5056","Type":"ContainerStarted","Data":"35dd00a1792753bdb31c52c5c9c32c89e1ef10eedb20d92e6755cbe041913401"}
Nov 03 00:18:49.648139 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:49.648077    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf" podStartSLOduration=2.648059443 podStartE2EDuration="2.648059443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:49.646891646 +0000 UTC m=+812.526475691" watchObservedRunningTime="2026-11-03 00:18:49.648059443 +0000 UTC m=+812.527643487"
Nov 03 00:18:49.712972 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:49.711953    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf" podStartSLOduration=2.711901225 podStartE2EDuration="2.711901225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:49.708503471 +0000 UTC m=+812.588087516" watchObservedRunningTime="2026-11-03 00:18:49.711901225 +0000 UTC m=+812.591485269"
Nov 03 00:18:49.769058 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:49.766448    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m" podStartSLOduration=2.766430871 podStartE2EDuration="2.766430871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:49.763968435 +0000 UTC m=+812.643552481" watchObservedRunningTime="2026-11-03 00:18:49.766430871 +0000 UTC m=+812.646014915"
Nov 03 00:18:50.213962 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:50.205613    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:50.213962 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:50.205742    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7fw2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 00:18:50.213962 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:50.205784    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m" podUID="188d122e-1edb-44d5-8c9c-0119c8bb4e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 00:18:50.231964 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:50.228867    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:50.231964 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:50.229072    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 00:18:50.231964 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:50.229123    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf" podUID="876fffce-7b9b-4aeb-8437-4c60987a5056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 00:18:50.264142 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:50.264093    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:50.264330 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:50.264240    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rgwqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 00:18:50.264330 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:50.264281    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf" podUID="19edcff3-65d5-489c-bb33-7ad5e4bcc791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 00:18:51.213043 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:51.213004    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7fw2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 00:18:51.214452 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:51.214419    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m" podUID="188d122e-1edb-44d5-8c9c-0119c8bb4e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 00:18:51.214970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:51.214345    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 00:18:51.216162 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:51.216137    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf" podUID="876fffce-7b9b-4aeb-8437-4c60987a5056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 00:18:51.257194 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:51.257158    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rgwqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 00:18:51.257437 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:51.257411    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf" podUID="19edcff3-65d5-489c-bb33-7ad5e4bcc791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 00:18:51.483674 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:51.483578    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx"
Nov 03 00:18:51.572379 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:51.571531    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c"]
Nov 03 00:18:52.204935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:52.204288    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7fw2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 00:18:52.204935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:52.204350    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m" podUID="188d122e-1edb-44d5-8c9c-0119c8bb4e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 00:18:52.216724 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:52.216688    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 00:18:52.217298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:52.217237    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf" podUID="876fffce-7b9b-4aeb-8437-4c60987a5056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 00:18:52.255934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:52.255372    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rgwqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 00:18:52.255934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:52.255440    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf" podUID="19edcff3-65d5-489c-bb33-7ad5e4bcc791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 00:18:53.204324 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:53.204286    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7fw2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 00:18:53.204728 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:53.204695    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m" podUID="188d122e-1edb-44d5-8c9c-0119c8bb4e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 00:18:53.212029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:53.212000    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 00:18:53.212257 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:53.212231    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf" podUID="876fffce-7b9b-4aeb-8437-4c60987a5056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 00:18:53.266938 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:53.266873    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rgwqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 00:18:53.268056 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:53.268023    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf" podUID="19edcff3-65d5-489c-bb33-7ad5e4bcc791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 00:18:54.207316 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:54.207275    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7fw2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 00:18:54.207595 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:54.207568    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m" podUID="188d122e-1edb-44d5-8c9c-0119c8bb4e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 00:18:54.228826 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:54.228789    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 00:18:54.229112 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:54.229085    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf" podUID="876fffce-7b9b-4aeb-8437-4c60987a5056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 00:18:54.256414 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:54.256377    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rgwqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 00:18:54.256673 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:54.256647    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf" podUID="19edcff3-65d5-489c-bb33-7ad5e4bcc791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 00:18:55.204061 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:55.204018    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7fw2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 00:18:55.204581 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:55.204091    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m" podUID="188d122e-1edb-44d5-8c9c-0119c8bb4e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 00:18:55.254814 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:55.254775    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rgwqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 00:18:55.255058 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:55.254840    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf" podUID="19edcff3-65d5-489c-bb33-7ad5e4bcc791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 00:18:55.458458 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:55.458377    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:55.459225 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:55.459196    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:55.460840 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:55.460600    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 00:18:56.204621 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:56.204574    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-7fw2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 00:18:56.205187 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:56.204638    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m" podUID="188d122e-1edb-44d5-8c9c-0119c8bb4e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 00:18:56.255344 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:56.255296    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rgwqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 00:18:56.255517 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:56.255376    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf" podUID="19edcff3-65d5-489c-bb33-7ad5e4bcc791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 00:18:56.572165 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:56.572038    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c" podUID="ec632772-d93a-4cfe-a905-433fc4ed7024" containerName="payload-processing" containerID="cri-o://3dfe60f793b603dfced9adcfc48015f271f95604c2529190eb4a9c700e8b6bbc" gracePeriod=25
Nov 03 00:18:56.829323 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:56.827528    5012 generic.go:356] "Generic (PLEG): container finished" podID="ec632772-d93a-4cfe-a905-433fc4ed7024" containerID="3dfe60f793b603dfced9adcfc48015f271f95604c2529190eb4a9c700e8b6bbc" exitCode=0
Nov 03 00:18:56.832405 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:56.831897    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c" event={"ID":"ec632772-d93a-4cfe-a905-433fc4ed7024","Type":"ContainerDied","Data":"3dfe60f793b603dfced9adcfc48015f271f95604c2529190eb4a9c700e8b6bbc"}
Nov 03 00:18:56.925016 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:56.924983    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c"
Nov 03 00:18:57.039019 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:57.038970    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec632772-d93a-4cfe-a905-433fc4ed7024-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec632772-d93a-4cfe-a905-433fc4ed7024-routing-overlay-volume\") pod \"ec632772-d93a-4cfe-a905-433fc4ed7024\" (UID: \"ec632772-d93a-4cfe-a905-433fc4ed7024\") "
Nov 03 00:18:57.039229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:57.039162    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec632772-d93a-4cfe-a905-433fc4ed7024-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec632772-d93a-4cfe-a905-433fc4ed7024-provider-credentials\") pod \"ec632772-d93a-4cfe-a905-433fc4ed7024\" (UID: \"ec632772-d93a-4cfe-a905-433fc4ed7024\") "
Nov 03 00:18:57.039229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:57.039217    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec632772-d93a-4cfe-a905-433fc4ed7024-kube-api-access-8phsq\" (UniqueName: \"kubernetes.io/projected/ec632772-d93a-4cfe-a905-433fc4ed7024-kube-api-access-8phsq\") pod \"ec632772-d93a-4cfe-a905-433fc4ed7024\" (UID: \"ec632772-d93a-4cfe-a905-433fc4ed7024\") "
Nov 03 00:18:57.039354 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:57.039286    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec632772-d93a-4cfe-a905-433fc4ed7024-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec632772-d93a-4cfe-a905-433fc4ed7024-plugins-config-volume\") pod \"ec632772-d93a-4cfe-a905-433fc4ed7024\" (UID: \"ec632772-d93a-4cfe-a905-433fc4ed7024\") "
Nov 03 00:18:57.039835 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:57.039800    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec632772-d93a-4cfe-a905-433fc4ed7024-routing-overlay-volume" pod "ec632772-d93a-4cfe-a905-433fc4ed7024" (UID: "ec632772-d93a-4cfe-a905-433fc4ed7024"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:18:57.040109 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:57.040080    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec632772-d93a-4cfe-a905-433fc4ed7024-plugins-config-volume" pod "ec632772-d93a-4cfe-a905-433fc4ed7024" (UID: "ec632772-d93a-4cfe-a905-433fc4ed7024"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:18:57.042182 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:57.042150    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec632772-d93a-4cfe-a905-433fc4ed7024-provider-credentials" pod "ec632772-d93a-4cfe-a905-433fc4ed7024" (UID: "ec632772-d93a-4cfe-a905-433fc4ed7024"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:18:57.042355 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:57.042326    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec632772-d93a-4cfe-a905-433fc4ed7024-kube-api-access-8phsq" pod "ec632772-d93a-4cfe-a905-433fc4ed7024" (UID: "ec632772-d93a-4cfe-a905-433fc4ed7024"). InnerVolumeSpecName "kube-api-access-8phsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:18:57.141503 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:57.141452    5012 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec632772-d93a-4cfe-a905-433fc4ed7024-provider-credentials\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:18:57.141503 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:57.141497    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8phsq\" (UniqueName: \"kubernetes.io/projected/ec632772-d93a-4cfe-a905-433fc4ed7024-kube-api-access-8phsq\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:18:57.141503 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:57.141514    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec632772-d93a-4cfe-a905-433fc4ed7024-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:18:57.141843 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:57.141528    5012 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec632772-d93a-4cfe-a905-433fc4ed7024-routing-overlay-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:18:57.474419 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:57.474324    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:57.474999 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:57.474934    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:57.476347 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:57.476311    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:18:57.507944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:57.503625    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:57.507944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:57.505355    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:57.507944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:57.506671    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:18:57.836168 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:57.836079    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c"
Nov 03 00:18:57.836168 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:57.836113    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c" event={"ID":"ec632772-d93a-4cfe-a905-433fc4ed7024","Type":"ContainerDied","Data":"3f23bc3b8b4396668e00d97ed44ec5684cad632257c1349430cbfe780421a0ab"}
Nov 03 00:18:57.836396 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:57.836177    5012 scope.go:122] "RemoveContainer" containerID="3dfe60f793b603dfced9adcfc48015f271f95604c2529190eb4a9c700e8b6bbc"
Nov 03 00:18:57.880487 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:57.880457    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c"]
Nov 03 00:18:57.883929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:57.883891    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-cdc4d9fcc-xrv2c"]
Nov 03 00:18:59.243404 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:18:59.243372    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec632772-d93a-4cfe-a905-433fc4ed7024" path="/var/lib/kubelet/pods/ec632772-d93a-4cfe-a905-433fc4ed7024/volumes"
Nov 03 00:19:02.545019 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:02.544984    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq"]
Nov 03 00:19:02.545758 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:02.545735    5012 cpu_manager.go:414] "containerMap: removing container" podUID="ec632772-d93a-4cfe-a905-433fc4ed7024" containerName="payload-processing"
Nov 03 00:19:02.545863 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:02.545760    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec632772-d93a-4cfe-a905-433fc4ed7024" containerName="payload-processing"
Nov 03 00:19:02.545944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:02.545897    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec632772-d93a-4cfe-a905-433fc4ed7024" containerName="payload-processing"
Nov 03 00:19:02.549342 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:02.549320    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq"
Nov 03 00:19:02.569317 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:02.569293    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq"]
Nov 03 00:19:02.574271 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:02.574183    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b39c24993f8f\""
Nov 03 00:19:02.603859 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:02.603829    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2jvj\" (UniqueName: \"kubernetes.io/projected/86738029-3783-4eef-8a7a-e043fb31eda1-kube-api-access-c2jvj\") pod \"payload-processing-external-model-6d94cc47fc-vkqbq\" (UID: \"86738029-3783-4eef-8a7a-e043fb31eda1\") " pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq"
Nov 03 00:19:02.603996 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:02.603956    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86738029-3783-4eef-8a7a-e043fb31eda1-provider-credentials\") pod \"payload-processing-external-model-6d94cc47fc-vkqbq\" (UID: \"86738029-3783-4eef-8a7a-e043fb31eda1\") " pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq"
Nov 03 00:19:02.604047 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:02.604019    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86738029-3783-4eef-8a7a-e043fb31eda1-routing-overlay-volume\") pod \"payload-processing-external-model-6d94cc47fc-vkqbq\" (UID: \"86738029-3783-4eef-8a7a-e043fb31eda1\") " pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq"
Nov 03 00:19:02.604247 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:02.604231    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86738029-3783-4eef-8a7a-e043fb31eda1-plugins-config-volume\") pod \"payload-processing-external-model-6d94cc47fc-vkqbq\" (UID: \"86738029-3783-4eef-8a7a-e043fb31eda1\") " pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq"
Nov 03 00:19:02.705088 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:02.705053    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86738029-3783-4eef-8a7a-e043fb31eda1-plugins-config-volume\") pod \"payload-processing-external-model-6d94cc47fc-vkqbq\" (UID: \"86738029-3783-4eef-8a7a-e043fb31eda1\") " pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq"
Nov 03 00:19:02.705255 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:02.705157    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2jvj\" (UniqueName: \"kubernetes.io/projected/86738029-3783-4eef-8a7a-e043fb31eda1-kube-api-access-c2jvj\") pod \"payload-processing-external-model-6d94cc47fc-vkqbq\" (UID: \"86738029-3783-4eef-8a7a-e043fb31eda1\") " pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq"
Nov 03 00:19:02.705255 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:02.705196    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86738029-3783-4eef-8a7a-e043fb31eda1-provider-credentials\") pod \"payload-processing-external-model-6d94cc47fc-vkqbq\" (UID: \"86738029-3783-4eef-8a7a-e043fb31eda1\") " pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq"
Nov 03 00:19:02.705255 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:02.705247    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86738029-3783-4eef-8a7a-e043fb31eda1-routing-overlay-volume\") pod \"payload-processing-external-model-6d94cc47fc-vkqbq\" (UID: \"86738029-3783-4eef-8a7a-e043fb31eda1\") " pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq"
Nov 03 00:19:02.705889 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:02.705860    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86738029-3783-4eef-8a7a-e043fb31eda1-routing-overlay-volume\") pod \"payload-processing-external-model-6d94cc47fc-vkqbq\" (UID: \"86738029-3783-4eef-8a7a-e043fb31eda1\") " pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq"
Nov 03 00:19:02.706014 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:02.705874    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86738029-3783-4eef-8a7a-e043fb31eda1-plugins-config-volume\") pod \"payload-processing-external-model-6d94cc47fc-vkqbq\" (UID: \"86738029-3783-4eef-8a7a-e043fb31eda1\") " pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq"
Nov 03 00:19:02.707948 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:02.707922    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86738029-3783-4eef-8a7a-e043fb31eda1-provider-credentials\") pod \"payload-processing-external-model-6d94cc47fc-vkqbq\" (UID: \"86738029-3783-4eef-8a7a-e043fb31eda1\") " pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq"
Nov 03 00:19:02.727352 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:02.727326    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2jvj\" (UniqueName: \"kubernetes.io/projected/86738029-3783-4eef-8a7a-e043fb31eda1-kube-api-access-c2jvj\") pod \"payload-processing-external-model-6d94cc47fc-vkqbq\" (UID: \"86738029-3783-4eef-8a7a-e043fb31eda1\") " pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq"
Nov 03 00:19:02.863485 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:02.863454    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq"
Nov 03 00:19:03.062964 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.062932    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"]
Nov 03 00:19:03.067876 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.067860    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.112803 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.112768    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"]
Nov 03 00:19:03.172119 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.172091    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq"]
Nov 03 00:19:03.175084 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:19:03.175054    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86738029_3783_4eef_8a7a_e043fb31eda1.slice/crio-d80cddcd560bd7011c7a7bc6abad2eea671feee6809282d00917f27c7f429b84 WatchSource:0}: Error finding container d80cddcd560bd7011c7a7bc6abad2eea671feee6809282d00917f27c7f429b84: Status 404 returned error can't find the container with id d80cddcd560bd7011c7a7bc6abad2eea671feee6809282d00917f27c7f429b84
Nov 03 00:19:03.214735 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.214708    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3684a36f-b227-463d-af7f-e8472ce6a45a-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.214852 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.214743    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.214852 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.214783    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.215000 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.214975    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.215057 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.215029    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.215151 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.215131    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.215215 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.215182    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.215286 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.215270    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.215353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.215333    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhbw9\" (UniqueName: \"kubernetes.io/projected/3684a36f-b227-463d-af7f-e8472ce6a45a-kube-api-access-vhbw9\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.316838 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.316804    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.317159 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.317131    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.317294 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.317184    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.317294 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.317260    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.317428 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.317304    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.317428 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.317345    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.317428 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.317392    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhbw9\" (UniqueName: \"kubernetes.io/projected/3684a36f-b227-463d-af7f-e8472ce6a45a-kube-api-access-vhbw9\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.317623 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.317532    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3684a36f-b227-463d-af7f-e8472ce6a45a-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.317623 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.317570    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.317881 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.317851    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.318018 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.317924    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.318979 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.318950    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.319131 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.319108    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3684a36f-b227-463d-af7f-e8472ce6a45a-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.319281 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.319256    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.320189 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.320166    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.320528 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.320506    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.336650 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.336607    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.337929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.337083    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhbw9\" (UniqueName: \"kubernetes.io/projected/3684a36f-b227-463d-af7f-e8472ce6a45a-kube-api-access-vhbw9\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.387393 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.387356    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:03.747857 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.747732    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"]
Nov 03 00:19:03.749145 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:19:03.749116    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3684a36f_b227_463d_af7f_e8472ce6a45a.slice/crio-a42111fe6e8415f850d6f2592326d24deedd0aefc3355c0c82acf9481296d799 WatchSource:0}: Error finding container a42111fe6e8415f850d6f2592326d24deedd0aefc3355c0c82acf9481296d799: Status 404 returned error can't find the container with id a42111fe6e8415f850d6f2592326d24deedd0aefc3355c0c82acf9481296d799
Nov 03 00:19:03.751523 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.751488    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:19:03.751618 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.751596    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:19:03.751692 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.751643    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:19:03.884738 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.884696    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57" event={"ID":"3684a36f-b227-463d-af7f-e8472ce6a45a","Type":"ContainerStarted","Data":"b537504792fcdd667e9c372fcf794ac237e3c79f6e05bdb2ef2023df5ae32602"}
Nov 03 00:19:03.884904 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.884773    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57" event={"ID":"3684a36f-b227-463d-af7f-e8472ce6a45a","Type":"ContainerStarted","Data":"a42111fe6e8415f850d6f2592326d24deedd0aefc3355c0c82acf9481296d799"}
Nov 03 00:19:03.887085 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.886957    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq" event={"ID":"86738029-3783-4eef-8a7a-e043fb31eda1","Type":"ContainerStarted","Data":"1bff2999abb52f4378611bfed41390383e8f38aee9a7d5845bc6354300f74b35"}
Nov 03 00:19:03.887085 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.887005    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq" event={"ID":"86738029-3783-4eef-8a7a-e043fb31eda1","Type":"ContainerStarted","Data":"d80cddcd560bd7011c7a7bc6abad2eea671feee6809282d00917f27c7f429b84"}
Nov 03 00:19:03.887085 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.887053    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq"
Nov 03 00:19:03.925930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.924222    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57" podStartSLOduration=0.924203286 podStartE2EDuration="924.203286ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:19:03.9224036 +0000 UTC m=+826.801987655" watchObservedRunningTime="2026-11-03 00:19:03.924203286 +0000 UTC m=+826.803787331"
Nov 03 00:19:03.963930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:03.963850    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq" podStartSLOduration=1.963831292 podStartE2EDuration="1.963831292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:19:03.961788267 +0000 UTC m=+826.841372319" watchObservedRunningTime="2026-11-03 00:19:03.963831292 +0000 UTC m=+826.843415336"
Nov 03 00:19:04.392933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:04.391260    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:04.410945 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:04.407064    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 00:19:04.410945 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:04.407129    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57" podUID="3684a36f-b227-463d-af7f-e8472ce6a45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 00:19:05.391948 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:05.391834    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 00:19:05.391948 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:05.391897    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57" podUID="3684a36f-b227-463d-af7f-e8472ce6a45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 00:19:06.392936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:06.390335    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 00:19:06.392936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:06.390402    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57" podUID="3684a36f-b227-463d-af7f-e8472ce6a45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 00:19:07.363939 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:19:07.363879    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:19:07.405944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:07.390391    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 00:19:07.405944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:07.390462    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57" podUID="3684a36f-b227-463d-af7f-e8472ce6a45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 00:19:08.408239 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:08.408200    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 00:19:08.408857 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:08.408826    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57" podUID="3684a36f-b227-463d-af7f-e8472ce6a45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 00:19:09.408936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:09.404312    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 00:19:09.408936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:09.404394    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57" podUID="3684a36f-b227-463d-af7f-e8472ce6a45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 00:19:10.733773 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:10.733733    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:10.734593 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:10.734554    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:10.735601 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:10.735577    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:19:14.937157 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:14.936085    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq"
Nov 03 00:19:15.197698 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:15.197612    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx"]
Nov 03 00:19:20.199221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:20.199143    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx" podUID="0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55" containerName="payload-processing" containerID="cri-o://3a69352c31554ff30543f00ef7f7e75fcc8dadfbeb977cf2c939cdada83bc4c6" gracePeriod=25
Nov 03 00:19:20.472587 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:20.472561    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx"
Nov 03 00:19:20.586373 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:20.586337    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-plugins-config-volume\") pod \"0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55\" (UID: \"0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55\") "
Nov 03 00:19:20.586373 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:20.586377    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-kube-api-access-9j55v\" (UniqueName: \"kubernetes.io/projected/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-kube-api-access-9j55v\") pod \"0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55\" (UID: \"0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55\") "
Nov 03 00:19:20.586601 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:20.586412    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-routing-overlay-volume\") pod \"0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55\" (UID: \"0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55\") "
Nov 03 00:19:20.586601 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:20.586487    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-provider-credentials\") pod \"0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55\" (UID: \"0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55\") "
Nov 03 00:19:20.586873 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:20.586849    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-plugins-config-volume" pod "0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55" (UID: "0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:19:20.586946 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:20.586887    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-routing-overlay-volume" pod "0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55" (UID: "0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:19:20.588548 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:20.588524    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-kube-api-access-9j55v" pod "0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55" (UID: "0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55"). InnerVolumeSpecName "kube-api-access-9j55v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:19:20.588680 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:20.588633    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-provider-credentials" pod "0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55" (UID: "0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:19:20.687741 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:20.687696    5012 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-provider-credentials\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:19:20.687741 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:20.687734    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:19:20.687741 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:20.687744    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j55v\" (UniqueName: \"kubernetes.io/projected/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-kube-api-access-9j55v\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:19:20.688043 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:20.687755    5012 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55-routing-overlay-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:19:21.234482 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:21.234442    5012 generic.go:356] "Generic (PLEG): container finished" podID="0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55" containerID="3a69352c31554ff30543f00ef7f7e75fcc8dadfbeb977cf2c939cdada83bc4c6" exitCode=0
Nov 03 00:19:21.234965 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:21.234495    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx"
Nov 03 00:19:21.234965 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:21.234514    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx" event={"ID":"0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55","Type":"ContainerDied","Data":"3a69352c31554ff30543f00ef7f7e75fcc8dadfbeb977cf2c939cdada83bc4c6"}
Nov 03 00:19:21.234965 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:21.234584    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx" event={"ID":"0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55","Type":"ContainerDied","Data":"1aa80c3376c3e8f63362994e33c072dbbc99b84a70b166fd8c64b2395159a5f2"}
Nov 03 00:19:21.234965 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:21.234609    5012 scope.go:122] "RemoveContainer" containerID="3a69352c31554ff30543f00ef7f7e75fcc8dadfbeb977cf2c939cdada83bc4c6"
Nov 03 00:19:21.254303 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:21.254275    5012 scope.go:122] "RemoveContainer" containerID="3a69352c31554ff30543f00ef7f7e75fcc8dadfbeb977cf2c939cdada83bc4c6"
Nov 03 00:19:21.254666 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:19:21.254644    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a69352c31554ff30543f00ef7f7e75fcc8dadfbeb977cf2c939cdada83bc4c6\": container with ID starting with 3a69352c31554ff30543f00ef7f7e75fcc8dadfbeb977cf2c939cdada83bc4c6 not found: ID does not exist" containerID="3a69352c31554ff30543f00ef7f7e75fcc8dadfbeb977cf2c939cdada83bc4c6"
Nov 03 00:19:21.254714 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:21.254675    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a69352c31554ff30543f00ef7f7e75fcc8dadfbeb977cf2c939cdada83bc4c6"} err="failed to get container status \"3a69352c31554ff30543f00ef7f7e75fcc8dadfbeb977cf2c939cdada83bc4c6\": rpc error: code = NotFound desc = could not find container \"3a69352c31554ff30543f00ef7f7e75fcc8dadfbeb977cf2c939cdada83bc4c6\": container with ID starting with 3a69352c31554ff30543f00ef7f7e75fcc8dadfbeb977cf2c939cdada83bc4c6 not found: ID does not exist"
Nov 03 00:19:21.306871 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:21.306829    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx"]
Nov 03 00:19:21.316780 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:21.316752    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-865949b6c4-vjfgx"]
Nov 03 00:19:23.244346 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:23.244311    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55" path="/var/lib/kubelet/pods/0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55/volumes"
Nov 03 00:19:28.125170 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:28.125039    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8"]
Nov 03 00:19:28.126073 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:28.126052    5012 cpu_manager.go:414] "containerMap: removing container" podUID="0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55" containerName="payload-processing"
Nov 03 00:19:28.126145 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:28.126076    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55" containerName="payload-processing"
Nov 03 00:19:28.126195 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:28.126185    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c579e4b-ed0b-4ed2-83f9-e644e7f1ee55" containerName="payload-processing"
Nov 03 00:19:28.131560 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:28.131537    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8"
Nov 03 00:19:28.146519 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:28.146486    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8"]
Nov 03 00:19:28.151298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:28.151265    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-18a113485b95\""
Nov 03 00:19:28.151298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:28.151265    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-10cbbedab37d\""
Nov 03 00:19:28.278062 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:28.278024    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e79035-3352-4932-801f-db7ced694849-plugins-config-volume\") pod \"payload-processing-external-model-f756c6b59-vm8d8\" (UID: \"73e79035-3352-4932-801f-db7ced694849\") " pod="models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8"
Nov 03 00:19:28.278258 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:28.278077    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/73e79035-3352-4932-801f-db7ced694849-routing-overlay-volume\") pod \"payload-processing-external-model-f756c6b59-vm8d8\" (UID: \"73e79035-3352-4932-801f-db7ced694849\") " pod="models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8"
Nov 03 00:19:28.278258 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:28.278186    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfnt4\" (UniqueName: \"kubernetes.io/projected/73e79035-3352-4932-801f-db7ced694849-kube-api-access-qfnt4\") pod \"payload-processing-external-model-f756c6b59-vm8d8\" (UID: \"73e79035-3352-4932-801f-db7ced694849\") " pod="models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8"
Nov 03 00:19:28.278258 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:28.278222    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/73e79035-3352-4932-801f-db7ced694849-provider-credentials\") pod \"payload-processing-external-model-f756c6b59-vm8d8\" (UID: \"73e79035-3352-4932-801f-db7ced694849\") " pod="models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8"
Nov 03 00:19:28.383456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:28.383002    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e79035-3352-4932-801f-db7ced694849-plugins-config-volume\") pod \"payload-processing-external-model-f756c6b59-vm8d8\" (UID: \"73e79035-3352-4932-801f-db7ced694849\") " pod="models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8"
Nov 03 00:19:28.383456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:28.383096    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/73e79035-3352-4932-801f-db7ced694849-routing-overlay-volume\") pod \"payload-processing-external-model-f756c6b59-vm8d8\" (UID: \"73e79035-3352-4932-801f-db7ced694849\") " pod="models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8"
Nov 03 00:19:28.383456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:28.383157    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfnt4\" (UniqueName: \"kubernetes.io/projected/73e79035-3352-4932-801f-db7ced694849-kube-api-access-qfnt4\") pod \"payload-processing-external-model-f756c6b59-vm8d8\" (UID: \"73e79035-3352-4932-801f-db7ced694849\") " pod="models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8"
Nov 03 00:19:28.383456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:28.383194    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/73e79035-3352-4932-801f-db7ced694849-provider-credentials\") pod \"payload-processing-external-model-f756c6b59-vm8d8\" (UID: \"73e79035-3352-4932-801f-db7ced694849\") " pod="models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8"
Nov 03 00:19:28.384589 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:28.384552    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/73e79035-3352-4932-801f-db7ced694849-routing-overlay-volume\") pod \"payload-processing-external-model-f756c6b59-vm8d8\" (UID: \"73e79035-3352-4932-801f-db7ced694849\") " pod="models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8"
Nov 03 00:19:28.386880 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:28.386204    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e79035-3352-4932-801f-db7ced694849-plugins-config-volume\") pod \"payload-processing-external-model-f756c6b59-vm8d8\" (UID: \"73e79035-3352-4932-801f-db7ced694849\") " pod="models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8"
Nov 03 00:19:28.387227 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:28.387199    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/73e79035-3352-4932-801f-db7ced694849-provider-credentials\") pod \"payload-processing-external-model-f756c6b59-vm8d8\" (UID: \"73e79035-3352-4932-801f-db7ced694849\") " pod="models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8"
Nov 03 00:19:28.409040 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:28.409004    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfnt4\" (UniqueName: \"kubernetes.io/projected/73e79035-3352-4932-801f-db7ced694849-kube-api-access-qfnt4\") pod \"payload-processing-external-model-f756c6b59-vm8d8\" (UID: \"73e79035-3352-4932-801f-db7ced694849\") " pod="models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8"
Nov 03 00:19:28.449377 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:28.449336    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8"
Nov 03 00:19:28.754674 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:28.754637    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8"]
Nov 03 00:19:28.755394 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:19:28.755367    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e79035_3352_4932_801f_db7ced694849.slice/crio-d6aa316a4aafbdca43864587e84024b43432c2fff5d82d3730fcad7815fc2e61 WatchSource:0}: Error finding container d6aa316a4aafbdca43864587e84024b43432c2fff5d82d3730fcad7815fc2e61: Status 404 returned error can't find the container with id d6aa316a4aafbdca43864587e84024b43432c2fff5d82d3730fcad7815fc2e61
Nov 03 00:19:29.303962 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:29.303927    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8" event={"ID":"73e79035-3352-4932-801f-db7ced694849","Type":"ContainerStarted","Data":"3e4512568950fe1b157ee2f35a69d793d2ff8449276279ddd617528c16ed2023"}
Nov 03 00:19:29.304421 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:29.303985    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8" event={"ID":"73e79035-3352-4932-801f-db7ced694849","Type":"ContainerStarted","Data":"d6aa316a4aafbdca43864587e84024b43432c2fff5d82d3730fcad7815fc2e61"}
Nov 03 00:19:29.304421 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:29.304061    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8"
Nov 03 00:19:29.339663 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:29.339608    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8" podStartSLOduration=1.339591145 podStartE2EDuration="1.339591145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:19:29.337276861 +0000 UTC m=+852.216860905" watchObservedRunningTime="2026-11-03 00:19:29.339591145 +0000 UTC m=+852.219175187"
Nov 03 00:19:32.998283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:32.998243    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"]
Nov 03 00:19:33.002418 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.002396    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.029390 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.029365    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"]
Nov 03 00:19:33.053090 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.053064    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8czxx\" (UniqueName: \"kubernetes.io/projected/5338078f-b28b-4f50-9594-a7a0647050a3-kube-api-access-8czxx\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.053239 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.053119    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.053239 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.053143    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.053239 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.053216    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5338078f-b28b-4f50-9594-a7a0647050a3-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.053402 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.053261    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.053402 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.053336    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5338078f-b28b-4f50-9594-a7a0647050a3-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.053523 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.053411    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.053523 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.053433    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.053523 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.053512    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5338078f-b28b-4f50-9594-a7a0647050a3-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.073009 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.072979    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"]
Nov 03 00:19:33.078138 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.078119    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.092773 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.092289    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-fc9316-openshift-default-dockercfg-8zk4q\""
Nov 03 00:19:33.098654 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.098627    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"]
Nov 03 00:19:33.218716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.163765    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.218716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.163823    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.218716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.163852    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5338078f-b28b-4f50-9594-a7a0647050a3-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.218716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.163881    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.218716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.163921    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee103686-3d06-4f4c-ab26-111525f4b2be-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.218716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.163955    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.218716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.164002    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.218716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.164058    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5338078f-b28b-4f50-9594-a7a0647050a3-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.218716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.164129    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.218716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.164165    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.218716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.164216    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.218716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.164254    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.218716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.164294    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.218716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.164341    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.218716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.164375    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.218716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.164415    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5338078f-b28b-4f50-9594-a7a0647050a3-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.218716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.164477    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8czxx\" (UniqueName: \"kubernetes.io/projected/5338078f-b28b-4f50-9594-a7a0647050a3-kube-api-access-8czxx\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.218716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.164572    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xhrh\" (UniqueName: \"kubernetes.io/projected/ee103686-3d06-4f4c-ab26-111525f4b2be-kube-api-access-9xhrh\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.218716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.165179    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.218716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.165436    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.218716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.166108    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5338078f-b28b-4f50-9594-a7a0647050a3-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.218716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.166407    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.218716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.217805    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.218716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.218561    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5338078f-b28b-4f50-9594-a7a0647050a3-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.227371 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.225362    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.266970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.265886    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.266970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.265966    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.266970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.266026    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.266970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.266064    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.266970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.266126    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.266970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.266217    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xhrh\" (UniqueName: \"kubernetes.io/projected/ee103686-3d06-4f4c-ab26-111525f4b2be-kube-api-access-9xhrh\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.266970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.266268    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.266970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.266292    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee103686-3d06-4f4c-ab26-111525f4b2be-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.266970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.266345    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.266970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.266750    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5338078f-b28b-4f50-9594-a7a0647050a3-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.303221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.267722    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.303221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.268358    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.303221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.268614    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.303221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.269314    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee103686-3d06-4f4c-ab26-111525f4b2be-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.303221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.272417    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.303221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.272651    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.303221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.275622    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8czxx\" (UniqueName: \"kubernetes.io/projected/5338078f-b28b-4f50-9594-a7a0647050a3-kube-api-access-8czxx\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.303221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.277156    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.303221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.289420    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.303221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.295635    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xhrh\" (UniqueName: \"kubernetes.io/projected/ee103686-3d06-4f4c-ab26-111525f4b2be-kube-api-access-9xhrh\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.318984 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.318942    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:33.399295 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.399018    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:33.766637 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.766588    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"]
Nov 03 00:19:33.782091 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.780591    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:19:33.782091 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.780686    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:19:33.782091 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.780729    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:19:33.999945 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:33.989808    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"]
Nov 03 00:19:34.040361 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:34.034151    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:19:34.040361 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:34.034236    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:19:34.040361 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:34.034281    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:19:34.450771 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:34.450730    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh" event={"ID":"ee103686-3d06-4f4c-ab26-111525f4b2be","Type":"ContainerStarted","Data":"f41f2f09efb258aca4732a1e1a84da6d98d913948079e4d9e25650fbc9dc744c"}
Nov 03 00:19:34.451103 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:34.451077    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh" event={"ID":"ee103686-3d06-4f4c-ab26-111525f4b2be","Type":"ContainerStarted","Data":"ef31f5253eec973b64a9702b6e02cd15b254fc7495385ca406071908e3b5ec58"}
Nov 03 00:19:34.499935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:34.489523    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx" event={"ID":"5338078f-b28b-4f50-9594-a7a0647050a3","Type":"ContainerStarted","Data":"63e282ca9c7b7a6f4f8c7ed1259839e0edff602d8a4439c5003a95fcc7f9a61b"}
Nov 03 00:19:34.499935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:34.489599    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx" event={"ID":"5338078f-b28b-4f50-9594-a7a0647050a3","Type":"ContainerStarted","Data":"c6e8ce2e01ab86ff04358f83ee66bc9cd2b3b7920f665ed1f09ea0800a7f5be3"}
Nov 03 00:19:34.575315 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:34.575251    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh" podStartSLOduration=1.575231911 podStartE2EDuration="1.575231911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:19:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:19:34.532614247 +0000 UTC m=+857.412198304" watchObservedRunningTime="2026-11-03 00:19:34.575231911 +0000 UTC m=+857.454815956"
Nov 03 00:19:34.579099 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:34.578964    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx" podStartSLOduration=2.578951245 podStartE2EDuration="2.578951245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:19:34.571402875 +0000 UTC m=+857.450986920" watchObservedRunningTime="2026-11-03 00:19:34.578951245 +0000 UTC m=+857.458535289"
Nov 03 00:19:35.327933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:35.321056    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:35.327933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:35.323372    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 00:19:35.327933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:35.323457    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx" podUID="5338078f-b28b-4f50-9594-a7a0647050a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 00:19:35.407384 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:35.405380    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:35.407700 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:35.407312    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 00:19:35.407931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:35.407876    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh" podUID="ee103686-3d06-4f4c-ab26-111525f4b2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 00:19:36.003935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:36.003444    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:19:36.320404 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:36.320308    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 00:19:36.320678 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:36.320645    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx" podUID="5338078f-b28b-4f50-9594-a7a0647050a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 00:19:36.417934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:36.416327    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:19:36.434933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:36.419440    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 00:19:36.434933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:36.419493    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh" podUID="ee103686-3d06-4f4c-ab26-111525f4b2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 00:19:36.553810 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:36.553762    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq" podUID="b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:19:36.956134 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:36.956088    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:19:37.005131 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:37.005085    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:19:37.006217 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:37.006195    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:19:37.347990 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:37.322512    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 00:19:37.348264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:37.348223    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx" podUID="5338078f-b28b-4f50-9594-a7a0647050a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 00:19:37.422065 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:37.422026    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 00:19:37.422649 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:37.422615    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh" podUID="ee103686-3d06-4f4c-ab26-111525f4b2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 00:19:37.423450 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:37.423419    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:19:37.449632 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:37.449601    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-5lnf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:19:37.449885 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:37.449857    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2" podUID="aa430510-e80c-4743-b4f3-838ce423e442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:19:37.477984 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:37.473177    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:19:37.556070 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:37.556014    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq" podUID="b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:19:37.557139 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:37.557117    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:19:37.835990 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:19:37.827106    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:19:37.965269 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:37.965224    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 03 00:19:37.966053 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:37.966030    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:19:37.968880 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:37.968860    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:19:38.005537 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:38.005499    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:19:38.104993 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:38.084677    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:19:38.147953 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:38.123546    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:19:38.163972 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:38.148616    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:19:38.344659 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:38.344624    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 00:19:38.344927 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:38.344885    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx" podUID="5338078f-b28b-4f50-9594-a7a0647050a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 00:19:38.415398 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:38.415304    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 00:19:38.415709 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:38.415677    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh" podUID="ee103686-3d06-4f4c-ab26-111525f4b2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 00:19:39.322052 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:39.322018    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 00:19:39.322784 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:39.322744    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx" podUID="5338078f-b28b-4f50-9594-a7a0647050a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 00:19:39.410934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:39.407236    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 00:19:39.410934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:39.407291    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh" podUID="ee103686-3d06-4f4c-ab26-111525f4b2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 00:19:40.321613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:40.319734    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 00:19:40.321613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:40.319791    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx" podUID="5338078f-b28b-4f50-9594-a7a0647050a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 00:19:40.335754 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:40.335728    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8"
Nov 03 00:19:40.446294 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:40.446246    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 00:19:40.446486 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:40.446325    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh" podUID="ee103686-3d06-4f4c-ab26-111525f4b2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 00:19:40.447700 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:40.447673    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq"]
Nov 03 00:19:41.321929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:41.321286    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 00:19:41.321929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:41.321348    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx" podUID="5338078f-b28b-4f50-9594-a7a0647050a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 00:19:41.409935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:41.409596    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 00:19:41.409935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:41.409679    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh" podUID="ee103686-3d06-4f4c-ab26-111525f4b2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 00:19:42.320302 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:42.320265    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 00:19:42.320500 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:42.320324    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx" podUID="5338078f-b28b-4f50-9594-a7a0647050a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 00:19:42.400643 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:42.400609    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 00:19:42.400955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:42.400895    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh" podUID="ee103686-3d06-4f4c-ab26-111525f4b2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 00:19:43.323538 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:43.323506    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:43.323964 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:43.323945    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:43.324607 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:43.324588    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:19:43.403664 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:43.403643    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:43.403948 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:43.403901    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:43.404722 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:43.404707    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:19:45.448273 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:45.448234    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq" podUID="86738029-3783-4eef-8a7a-e043fb31eda1" containerName="payload-processing" containerID="cri-o://1bff2999abb52f4378611bfed41390383e8f38aee9a7d5845bc6354300f74b35" gracePeriod=25
Nov 03 00:19:45.732954 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:45.732921    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq"
Nov 03 00:19:45.791847 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:45.791816    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86738029-3783-4eef-8a7a-e043fb31eda1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86738029-3783-4eef-8a7a-e043fb31eda1-routing-overlay-volume\") pod \"86738029-3783-4eef-8a7a-e043fb31eda1\" (UID: \"86738029-3783-4eef-8a7a-e043fb31eda1\") "
Nov 03 00:19:45.792029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:45.791936    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86738029-3783-4eef-8a7a-e043fb31eda1-kube-api-access-c2jvj\" (UniqueName: \"kubernetes.io/projected/86738029-3783-4eef-8a7a-e043fb31eda1-kube-api-access-c2jvj\") pod \"86738029-3783-4eef-8a7a-e043fb31eda1\" (UID: \"86738029-3783-4eef-8a7a-e043fb31eda1\") "
Nov 03 00:19:45.792029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:45.791989    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86738029-3783-4eef-8a7a-e043fb31eda1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86738029-3783-4eef-8a7a-e043fb31eda1-plugins-config-volume\") pod \"86738029-3783-4eef-8a7a-e043fb31eda1\" (UID: \"86738029-3783-4eef-8a7a-e043fb31eda1\") "
Nov 03 00:19:45.792150 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:45.792099    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86738029-3783-4eef-8a7a-e043fb31eda1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/86738029-3783-4eef-8a7a-e043fb31eda1-provider-credentials\") pod \"86738029-3783-4eef-8a7a-e043fb31eda1\" (UID: \"86738029-3783-4eef-8a7a-e043fb31eda1\") "
Nov 03 00:19:45.792341 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:45.792311    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86738029-3783-4eef-8a7a-e043fb31eda1-routing-overlay-volume" pod "86738029-3783-4eef-8a7a-e043fb31eda1" (UID: "86738029-3783-4eef-8a7a-e043fb31eda1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:19:45.792421 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:45.792395    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86738029-3783-4eef-8a7a-e043fb31eda1-plugins-config-volume" pod "86738029-3783-4eef-8a7a-e043fb31eda1" (UID: "86738029-3783-4eef-8a7a-e043fb31eda1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:19:45.792664 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:45.792642    5012 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86738029-3783-4eef-8a7a-e043fb31eda1-routing-overlay-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:19:45.792743 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:45.792668    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86738029-3783-4eef-8a7a-e043fb31eda1-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:19:45.794070 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:45.794049    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86738029-3783-4eef-8a7a-e043fb31eda1-provider-credentials" pod "86738029-3783-4eef-8a7a-e043fb31eda1" (UID: "86738029-3783-4eef-8a7a-e043fb31eda1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:19:45.794178 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:45.794153    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86738029-3783-4eef-8a7a-e043fb31eda1-kube-api-access-c2jvj" pod "86738029-3783-4eef-8a7a-e043fb31eda1" (UID: "86738029-3783-4eef-8a7a-e043fb31eda1"). InnerVolumeSpecName "kube-api-access-c2jvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:19:45.893837 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:45.893807    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2jvj\" (UniqueName: \"kubernetes.io/projected/86738029-3783-4eef-8a7a-e043fb31eda1-kube-api-access-c2jvj\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:19:45.893837 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:45.893834    5012 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86738029-3783-4eef-8a7a-e043fb31eda1-provider-credentials\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:19:46.025112 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:46.025015    5012 generic.go:356] "Generic (PLEG): container finished" podID="86738029-3783-4eef-8a7a-e043fb31eda1" containerID="1bff2999abb52f4378611bfed41390383e8f38aee9a7d5845bc6354300f74b35" exitCode=0
Nov 03 00:19:46.025283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:46.025169    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq" event={"ID":"86738029-3783-4eef-8a7a-e043fb31eda1","Type":"ContainerDied","Data":"1bff2999abb52f4378611bfed41390383e8f38aee9a7d5845bc6354300f74b35"}
Nov 03 00:19:46.025283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:46.025261    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq" event={"ID":"86738029-3783-4eef-8a7a-e043fb31eda1","Type":"ContainerDied","Data":"d80cddcd560bd7011c7a7bc6abad2eea671feee6809282d00917f27c7f429b84"}
Nov 03 00:19:46.025400 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:46.025298    5012 scope.go:122] "RemoveContainer" containerID="1bff2999abb52f4378611bfed41390383e8f38aee9a7d5845bc6354300f74b35"
Nov 03 00:19:46.025515 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:46.025496    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq"
Nov 03 00:19:46.041246 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:46.041220    5012 scope.go:122] "RemoveContainer" containerID="1bff2999abb52f4378611bfed41390383e8f38aee9a7d5845bc6354300f74b35"
Nov 03 00:19:46.041577 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:19:46.041549    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bff2999abb52f4378611bfed41390383e8f38aee9a7d5845bc6354300f74b35\": container with ID starting with 1bff2999abb52f4378611bfed41390383e8f38aee9a7d5845bc6354300f74b35 not found: ID does not exist" containerID="1bff2999abb52f4378611bfed41390383e8f38aee9a7d5845bc6354300f74b35"
Nov 03 00:19:46.041654 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:46.041587    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bff2999abb52f4378611bfed41390383e8f38aee9a7d5845bc6354300f74b35"} err="failed to get container status \"1bff2999abb52f4378611bfed41390383e8f38aee9a7d5845bc6354300f74b35\": rpc error: code = NotFound desc = could not find container \"1bff2999abb52f4378611bfed41390383e8f38aee9a7d5845bc6354300f74b35\": container with ID starting with 1bff2999abb52f4378611bfed41390383e8f38aee9a7d5845bc6354300f74b35 not found: ID does not exist"
Nov 03 00:19:46.070904 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:46.070874    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq"]
Nov 03 00:19:46.080606 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:46.080581    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d94cc47fc-vkqbq"]
Nov 03 00:19:46.635109 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:46.635073    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"]
Nov 03 00:19:47.256581 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:47.256547    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86738029-3783-4eef-8a7a-e043fb31eda1" path="/var/lib/kubelet/pods/86738029-3783-4eef-8a7a-e043fb31eda1/volumes"
Nov 03 00:19:48.979934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:48.979714    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:19:49.106029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:49.100987    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:19:49.129646 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:49.129612    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq" podUID="b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:19:49.310985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:49.302999    5012 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:19:49.764936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:49.746162    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-8fzrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:19:49.764936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:49.746302    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc" podUID="51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:19:50.272935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:50.270038    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:19:58.487482 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:58.487441    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-885c6809-mms4m"]
Nov 03 00:19:58.488147 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:58.488129    5012 cpu_manager.go:414] "containerMap: removing container" podUID="86738029-3783-4eef-8a7a-e043fb31eda1" containerName="payload-processing"
Nov 03 00:19:58.488219 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:58.488148    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86738029-3783-4eef-8a7a-e043fb31eda1" containerName="payload-processing"
Nov 03 00:19:58.488268 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:58.488258    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="86738029-3783-4eef-8a7a-e043fb31eda1" containerName="payload-processing"
Nov 03 00:19:58.493225 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:58.493199    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-885c6809-mms4m"
Nov 03 00:19:58.508364 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:58.508334    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-885c6809-mms4m"]
Nov 03 00:19:58.513170 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:58.513141    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 00:19:58.513170 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:58.513144    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:19:58.513459 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:58.513188    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 00:19:58.515120 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:58.515069    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:19:58.560306 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:58.560253    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/354fe5c6-af53-4068-8cf4-48d8fa63c611-maas-api-xks-ca\") pod \"maas-api-revoke-sub-885c6809-mms4m\" (UID: \"354fe5c6-af53-4068-8cf4-48d8fa63c611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885c6809-mms4m"
Nov 03 00:19:58.560558 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:58.560528    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lggz6\" (UniqueName: \"kubernetes.io/projected/354fe5c6-af53-4068-8cf4-48d8fa63c611-kube-api-access-lggz6\") pod \"maas-api-revoke-sub-885c6809-mms4m\" (UID: \"354fe5c6-af53-4068-8cf4-48d8fa63c611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885c6809-mms4m"
Nov 03 00:19:58.560610 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:58.560573    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/354fe5c6-af53-4068-8cf4-48d8fa63c611-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-885c6809-mms4m\" (UID: \"354fe5c6-af53-4068-8cf4-48d8fa63c611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885c6809-mms4m"
Nov 03 00:19:58.661589 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:58.661545    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lggz6\" (UniqueName: \"kubernetes.io/projected/354fe5c6-af53-4068-8cf4-48d8fa63c611-kube-api-access-lggz6\") pod \"maas-api-revoke-sub-885c6809-mms4m\" (UID: \"354fe5c6-af53-4068-8cf4-48d8fa63c611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885c6809-mms4m"
Nov 03 00:19:58.661589 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:58.661595    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/354fe5c6-af53-4068-8cf4-48d8fa63c611-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-885c6809-mms4m\" (UID: \"354fe5c6-af53-4068-8cf4-48d8fa63c611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885c6809-mms4m"
Nov 03 00:19:58.661826 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:58.661722    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/354fe5c6-af53-4068-8cf4-48d8fa63c611-maas-api-xks-ca\") pod \"maas-api-revoke-sub-885c6809-mms4m\" (UID: \"354fe5c6-af53-4068-8cf4-48d8fa63c611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885c6809-mms4m"
Nov 03 00:19:58.662376 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:58.662352    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/354fe5c6-af53-4068-8cf4-48d8fa63c611-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-885c6809-mms4m\" (UID: \"354fe5c6-af53-4068-8cf4-48d8fa63c611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885c6809-mms4m"
Nov 03 00:19:58.664127 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:58.664107    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/354fe5c6-af53-4068-8cf4-48d8fa63c611-maas-api-xks-ca\") pod \"maas-api-revoke-sub-885c6809-mms4m\" (UID: \"354fe5c6-af53-4068-8cf4-48d8fa63c611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885c6809-mms4m"
Nov 03 00:19:58.678638 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:58.678603    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lggz6\" (UniqueName: \"kubernetes.io/projected/354fe5c6-af53-4068-8cf4-48d8fa63c611-kube-api-access-lggz6\") pod \"maas-api-revoke-sub-885c6809-mms4m\" (UID: \"354fe5c6-af53-4068-8cf4-48d8fa63c611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885c6809-mms4m"
Nov 03 00:19:58.808425 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:58.808300    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-885c6809-mms4m"
Nov 03 00:19:59.112500 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:59.112463    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-885c6809-mms4m"]
Nov 03 00:19:59.113086 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:19:59.113047    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354fe5c6_af53_4068_8cf4_48d8fa63c611.slice/crio-b5171832c9447463ab74e41927400e203846b8fa4ecc5cc9b69ca0cd17983fae WatchSource:0}: Error finding container b5171832c9447463ab74e41927400e203846b8fa4ecc5cc9b69ca0cd17983fae: Status 404 returned error can't find the container with id b5171832c9447463ab74e41927400e203846b8fa4ecc5cc9b69ca0cd17983fae
Nov 03 00:19:59.235541 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:19:59.235497    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-885c6809-mms4m" event={"ID":"354fe5c6-af53-4068-8cf4-48d8fa63c611","Type":"ContainerStarted","Data":"b5171832c9447463ab74e41927400e203846b8fa4ecc5cc9b69ca0cd17983fae"}
Nov 03 00:20:00.245348 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:00.245306    5012 generic.go:356] "Generic (PLEG): container finished" podID="354fe5c6-af53-4068-8cf4-48d8fa63c611" containerID="1797669846dc938d1c04fb21a415cad84301ffaaf903c12db10551f45ffca9a3" exitCode=0
Nov 03 00:20:00.245794 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:00.245369    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-885c6809-mms4m" event={"ID":"354fe5c6-af53-4068-8cf4-48d8fa63c611","Type":"ContainerDied","Data":"1797669846dc938d1c04fb21a415cad84301ffaaf903c12db10551f45ffca9a3"}
Nov 03 00:20:01.473563 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:01.473532    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-885c6809-mms4m"
Nov 03 00:20:01.497054 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:01.497023    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/354fe5c6-af53-4068-8cf4-48d8fa63c611-kube-api-access-lggz6\" (UniqueName: \"kubernetes.io/projected/354fe5c6-af53-4068-8cf4-48d8fa63c611-kube-api-access-lggz6\") pod \"354fe5c6-af53-4068-8cf4-48d8fa63c611\" (UID: \"354fe5c6-af53-4068-8cf4-48d8fa63c611\") "
Nov 03 00:20:01.497242 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:01.497065    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/354fe5c6-af53-4068-8cf4-48d8fa63c611-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/354fe5c6-af53-4068-8cf4-48d8fa63c611-maas-api-openshift-ca\") pod \"354fe5c6-af53-4068-8cf4-48d8fa63c611\" (UID: \"354fe5c6-af53-4068-8cf4-48d8fa63c611\") "
Nov 03 00:20:01.497242 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:01.497090    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/354fe5c6-af53-4068-8cf4-48d8fa63c611-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/354fe5c6-af53-4068-8cf4-48d8fa63c611-maas-api-xks-ca\") pod \"354fe5c6-af53-4068-8cf4-48d8fa63c611\" (UID: \"354fe5c6-af53-4068-8cf4-48d8fa63c611\") "
Nov 03 00:20:01.497512 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:01.497483    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/354fe5c6-af53-4068-8cf4-48d8fa63c611-maas-api-openshift-ca" pod "354fe5c6-af53-4068-8cf4-48d8fa63c611" (UID: "354fe5c6-af53-4068-8cf4-48d8fa63c611"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:20:01.499261 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:01.499233    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/354fe5c6-af53-4068-8cf4-48d8fa63c611-maas-api-xks-ca" pod "354fe5c6-af53-4068-8cf4-48d8fa63c611" (UID: "354fe5c6-af53-4068-8cf4-48d8fa63c611"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:20:01.499385 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:01.499285    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354fe5c6-af53-4068-8cf4-48d8fa63c611-kube-api-access-lggz6" pod "354fe5c6-af53-4068-8cf4-48d8fa63c611" (UID: "354fe5c6-af53-4068-8cf4-48d8fa63c611"). InnerVolumeSpecName "kube-api-access-lggz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:20:01.598551 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:01.598498    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lggz6\" (UniqueName: \"kubernetes.io/projected/354fe5c6-af53-4068-8cf4-48d8fa63c611-kube-api-access-lggz6\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:01.598551 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:01.598535    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/354fe5c6-af53-4068-8cf4-48d8fa63c611-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:01.598551 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:01.598561    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/354fe5c6-af53-4068-8cf4-48d8fa63c611-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:01.657857 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:01.657813    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq" podUID="b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b" containerName="main" containerID="cri-o://66f5bbca51de8c2d3a4492897ce8539e5a9d897720a2d7f68b732b2acf3a320f" gracePeriod=45
Nov 03 00:20:01.967001 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:01.966968    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:20:02.003727 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.003693    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-tls-certs\" (UniqueName: \"kubernetes.io/secret/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-tls-certs\") pod \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\" (UID: \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\") "
Nov 03 00:20:02.003985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.003752    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-dshm\") pod \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\" (UID: \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\") "
Nov 03 00:20:02.003985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.003861    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-kube-api-access-9d6rk\" (UniqueName: \"kubernetes.io/projected/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-kube-api-access-9d6rk\") pod \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\" (UID: \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\") "
Nov 03 00:20:02.003985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.003953    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-home\" (UniqueName: \"kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-home\") pod \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\" (UID: \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\") "
Nov 03 00:20:02.004200 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.004106    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-tmp-dir\") pod \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\" (UID: \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\") "
Nov 03 00:20:02.004200 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.004138    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-model-cache\") pod \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\" (UID: \"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b\") "
Nov 03 00:20:02.004682 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.004506    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-tmp-dir" pod "b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b" (UID: "b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:20:02.004682 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.004593    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-home" pod "b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b" (UID: "b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:20:02.004682 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.004649    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-model-cache" pod "b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b" (UID: "b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:20:02.004887 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.004756    5012 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-home\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:02.004887 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.004775    5012 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-tmp-dir\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:02.004887 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.004789    5012 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-model-cache\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:02.007023 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.006984    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-dshm" pod "b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b" (UID: "b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:20:02.010934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.007261    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-kube-api-access-9d6rk" pod "b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b" (UID: "b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b"). InnerVolumeSpecName "kube-api-access-9d6rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:20:02.010934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.007276    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-tls-certs" pod "b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b" (UID: "b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:20:02.105567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.105529    5012 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-tls-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:02.105567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.105564    5012 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-dshm\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:02.105567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.105574    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d6rk\" (UniqueName: \"kubernetes.io/projected/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b-kube-api-access-9d6rk\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:02.289685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.289615    5012 generic.go:356] "Generic (PLEG): container finished" podID="b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b" containerID="66f5bbca51de8c2d3a4492897ce8539e5a9d897720a2d7f68b732b2acf3a320f" exitCode=0
Nov 03 00:20:02.289685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.289669    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq" event={"ID":"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b","Type":"ContainerDied","Data":"66f5bbca51de8c2d3a4492897ce8539e5a9d897720a2d7f68b732b2acf3a320f"}
Nov 03 00:20:02.289968 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.289736    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"
Nov 03 00:20:02.289968 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.289790    5012 scope.go:122] "RemoveContainer" containerID="66f5bbca51de8c2d3a4492897ce8539e5a9d897720a2d7f68b732b2acf3a320f"
Nov 03 00:20:02.289968 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.289952    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq" event={"ID":"b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b","Type":"ContainerDied","Data":"f07ad87faa14975348c9ffa1a3f26c2646d6e211f3d6dec4803b3e8158994334"}
Nov 03 00:20:02.292407 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.292381    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-885c6809-mms4m" event={"ID":"354fe5c6-af53-4068-8cf4-48d8fa63c611","Type":"ContainerDied","Data":"b5171832c9447463ab74e41927400e203846b8fa4ecc5cc9b69ca0cd17983fae"}
Nov 03 00:20:02.292521 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.292424    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-885c6809-mms4m"
Nov 03 00:20:02.292521 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.292461    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5171832c9447463ab74e41927400e203846b8fa4ecc5cc9b69ca0cd17983fae"
Nov 03 00:20:02.306424 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.306395    5012 scope.go:122] "RemoveContainer" containerID="66f5bbca51de8c2d3a4492897ce8539e5a9d897720a2d7f68b732b2acf3a320f"
Nov 03 00:20:02.306824 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:20:02.306795    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66f5bbca51de8c2d3a4492897ce8539e5a9d897720a2d7f68b732b2acf3a320f\": container with ID starting with 66f5bbca51de8c2d3a4492897ce8539e5a9d897720a2d7f68b732b2acf3a320f not found: ID does not exist" containerID="66f5bbca51de8c2d3a4492897ce8539e5a9d897720a2d7f68b732b2acf3a320f"
Nov 03 00:20:02.306950 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.306837    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66f5bbca51de8c2d3a4492897ce8539e5a9d897720a2d7f68b732b2acf3a320f"} err="failed to get container status \"66f5bbca51de8c2d3a4492897ce8539e5a9d897720a2d7f68b732b2acf3a320f\": rpc error: code = NotFound desc = could not find container \"66f5bbca51de8c2d3a4492897ce8539e5a9d897720a2d7f68b732b2acf3a320f\": container with ID starting with 66f5bbca51de8c2d3a4492897ce8539e5a9d897720a2d7f68b732b2acf3a320f not found: ID does not exist"
Nov 03 00:20:02.335183 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.335144    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"]
Nov 03 00:20:02.344373 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:02.344337    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-a-5f6115a4-kserve-744b669f7b-c4wvq"]
Nov 03 00:20:03.103334 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.103292    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"]
Nov 03 00:20:03.103878 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.103864    5012 cpu_manager.go:414] "containerMap: removing container" podUID="354fe5c6-af53-4068-8cf4-48d8fa63c611" containerName="revoke-keys"
Nov 03 00:20:03.103938 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.103879    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="354fe5c6-af53-4068-8cf4-48d8fa63c611" containerName="revoke-keys"
Nov 03 00:20:03.103978 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.103942    5012 cpu_manager.go:414] "containerMap: removing container" podUID="b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b" containerName="main"
Nov 03 00:20:03.103978 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.103948    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b" containerName="main"
Nov 03 00:20:03.104043 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.104029    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="354fe5c6-af53-4068-8cf4-48d8fa63c611" containerName="revoke-keys"
Nov 03 00:20:03.104043 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.104039    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b" containerName="main"
Nov 03 00:20:03.109195 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.109168    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.135129 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.135078    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"]
Nov 03 00:20:03.218717 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.218666    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.218717 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.218717    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvz2s\" (UniqueName: \"kubernetes.io/projected/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-kube-api-access-jvz2s\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.219027 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.218750    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.219027 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.218782    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.219027 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.218964    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.219135 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.219070    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.219135 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.219119    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.219235 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.219220    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.219316 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.219299    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.245062 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.245024    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b" path="/var/lib/kubelet/pods/b53f3e6b-ee4f-4fbc-8d8e-09b783764c4b/volumes"
Nov 03 00:20:03.320791 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.320742    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.321019 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.320844    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.321019 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.320896    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.321019 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.320937    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvz2s\" (UniqueName: \"kubernetes.io/projected/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-kube-api-access-jvz2s\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.321019 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.320972    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.321019 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.320995    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.321299 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.321036    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.321299 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.321109    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.321299 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.321139    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.321461 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.321403    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.321592 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.321564    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.321669 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.321653    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.321794 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.321772    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.321879 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.321861    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.324003 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.323980    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.324133 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.324119    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.338196 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.338162    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.338341 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.338304    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvz2s\" (UniqueName: \"kubernetes.io/projected/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-kube-api-access-jvz2s\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.426147 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.426054    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:03.748443 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.748403    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"]
Nov 03 00:20:03.748759 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:20:03.748731    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba16a59b_3dc5_4c8e_8378_70b27bdd0590.slice/crio-ce2287d07510405faca5b60fd06561f1482f851cb287cb7cbefe4e9e65ecd24f WatchSource:0}: Error finding container ce2287d07510405faca5b60fd06561f1482f851cb287cb7cbefe4e9e65ecd24f: Status 404 returned error can't find the container with id ce2287d07510405faca5b60fd06561f1482f851cb287cb7cbefe4e9e65ecd24f
Nov 03 00:20:03.751640 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.751589    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:03.751752 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.751699    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:03.751811 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:03.751794    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:04.311882 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:04.311838    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5" event={"ID":"ba16a59b-3dc5-4c8e-8378-70b27bdd0590","Type":"ContainerStarted","Data":"e5089a9404e1269ea4efbb75952ceeba291c3137ee3f0b1c6545eacac2b572d6"}
Nov 03 00:20:04.312358 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:04.311923    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5" event={"ID":"ba16a59b-3dc5-4c8e-8378-70b27bdd0590","Type":"ContainerStarted","Data":"ce2287d07510405faca5b60fd06561f1482f851cb287cb7cbefe4e9e65ecd24f"}
Nov 03 00:20:04.354948 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:04.354835    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5" podStartSLOduration=1.354815641 podStartE2EDuration="1.354815641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:04.354459704 +0000 UTC m=+887.234043747" watchObservedRunningTime="2026-11-03 00:20:04.354815641 +0000 UTC m=+887.234399685"
Nov 03 00:20:04.427259 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:04.427219    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:04.429254 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:04.429223    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 00:20:04.429407 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:04.429277    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5" podUID="ba16a59b-3dc5-4c8e-8378-70b27bdd0590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 00:20:05.427484 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:05.427433    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 00:20:05.427953 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:05.427530    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5" podUID="ba16a59b-3dc5-4c8e-8378-70b27bdd0590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 00:20:06.429451 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:06.429406    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 00:20:06.430134 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:06.430094    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5" podUID="ba16a59b-3dc5-4c8e-8378-70b27bdd0590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 00:20:07.363812 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:20:07.363776    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:20:07.426929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:07.426880    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 00:20:07.427100 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:07.426955    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5" podUID="ba16a59b-3dc5-4c8e-8378-70b27bdd0590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 00:20:08.426542 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:08.426503    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 00:20:08.426965 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:08.426559    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5" podUID="ba16a59b-3dc5-4c8e-8378-70b27bdd0590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 00:20:09.427109 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:09.427066    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 00:20:09.427556 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:09.427126    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5" podUID="ba16a59b-3dc5-4c8e-8378-70b27bdd0590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 00:20:10.306953 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.306903    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p"]
Nov 03 00:20:10.312451 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.312423    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p"
Nov 03 00:20:10.324863 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.324838    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p"]
Nov 03 00:20:10.355894 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.355858    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c36d8715-10c0-4e43-9ae1-e33445533437-routing-overlay-volume\") pod \"payload-processing-external-model-6d94cc47fc-4tc2p\" (UID: \"c36d8715-10c0-4e43-9ae1-e33445533437\") " pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p"
Nov 03 00:20:10.356164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.355935    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c36d8715-10c0-4e43-9ae1-e33445533437-plugins-config-volume\") pod \"payload-processing-external-model-6d94cc47fc-4tc2p\" (UID: \"c36d8715-10c0-4e43-9ae1-e33445533437\") " pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p"
Nov 03 00:20:10.356164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.356102    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flkz2\" (UniqueName: \"kubernetes.io/projected/c36d8715-10c0-4e43-9ae1-e33445533437-kube-api-access-flkz2\") pod \"payload-processing-external-model-6d94cc47fc-4tc2p\" (UID: \"c36d8715-10c0-4e43-9ae1-e33445533437\") " pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p"
Nov 03 00:20:10.356354 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.356336    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c36d8715-10c0-4e43-9ae1-e33445533437-provider-credentials\") pod \"payload-processing-external-model-6d94cc47fc-4tc2p\" (UID: \"c36d8715-10c0-4e43-9ae1-e33445533437\") " pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p"
Nov 03 00:20:10.427437 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.427406    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 00:20:10.427774 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.427462    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5" podUID="ba16a59b-3dc5-4c8e-8378-70b27bdd0590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 00:20:10.457812 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.457776    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c36d8715-10c0-4e43-9ae1-e33445533437-provider-credentials\") pod \"payload-processing-external-model-6d94cc47fc-4tc2p\" (UID: \"c36d8715-10c0-4e43-9ae1-e33445533437\") " pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p"
Nov 03 00:20:10.457997 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.457869    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c36d8715-10c0-4e43-9ae1-e33445533437-routing-overlay-volume\") pod \"payload-processing-external-model-6d94cc47fc-4tc2p\" (UID: \"c36d8715-10c0-4e43-9ae1-e33445533437\") " pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p"
Nov 03 00:20:10.457997 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.457942    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c36d8715-10c0-4e43-9ae1-e33445533437-plugins-config-volume\") pod \"payload-processing-external-model-6d94cc47fc-4tc2p\" (UID: \"c36d8715-10c0-4e43-9ae1-e33445533437\") " pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p"
Nov 03 00:20:10.458166 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.458007    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flkz2\" (UniqueName: \"kubernetes.io/projected/c36d8715-10c0-4e43-9ae1-e33445533437-kube-api-access-flkz2\") pod \"payload-processing-external-model-6d94cc47fc-4tc2p\" (UID: \"c36d8715-10c0-4e43-9ae1-e33445533437\") " pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p"
Nov 03 00:20:10.458622 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.458591    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c36d8715-10c0-4e43-9ae1-e33445533437-routing-overlay-volume\") pod \"payload-processing-external-model-6d94cc47fc-4tc2p\" (UID: \"c36d8715-10c0-4e43-9ae1-e33445533437\") " pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p"
Nov 03 00:20:10.458763 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.458741    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c36d8715-10c0-4e43-9ae1-e33445533437-plugins-config-volume\") pod \"payload-processing-external-model-6d94cc47fc-4tc2p\" (UID: \"c36d8715-10c0-4e43-9ae1-e33445533437\") " pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p"
Nov 03 00:20:10.460285 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.460256    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c36d8715-10c0-4e43-9ae1-e33445533437-provider-credentials\") pod \"payload-processing-external-model-6d94cc47fc-4tc2p\" (UID: \"c36d8715-10c0-4e43-9ae1-e33445533437\") " pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p"
Nov 03 00:20:10.475217 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.475190    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flkz2\" (UniqueName: \"kubernetes.io/projected/c36d8715-10c0-4e43-9ae1-e33445533437-kube-api-access-flkz2\") pod \"payload-processing-external-model-6d94cc47fc-4tc2p\" (UID: \"c36d8715-10c0-4e43-9ae1-e33445533437\") " pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p"
Nov 03 00:20:10.645948 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.628133    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p"
Nov 03 00:20:10.929812 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.929718    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4da3e6a9-dgsvs"]
Nov 03 00:20:10.934853 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.934828    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da3e6a9-dgsvs"
Nov 03 00:20:10.949360 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.949334    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4da3e6a9-dgsvs"]
Nov 03 00:20:10.953990 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.953963    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:20:10.954137 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.954016    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 00:20:10.958313 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.956297    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 00:20:10.958313 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.956364    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:20:10.989403 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:10.989368    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p"]
Nov 03 00:20:10.997598 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:20:10.997568    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc36d8715_10c0_4e43_9ae1_e33445533437.slice/crio-14d2f69ccf88fb864412a351a1e91e6ce05b679ddffab565518e6afef4490d3e WatchSource:0}: Error finding container 14d2f69ccf88fb864412a351a1e91e6ce05b679ddffab565518e6afef4490d3e: Status 404 returned error can't find the container with id 14d2f69ccf88fb864412a351a1e91e6ce05b679ddffab565518e6afef4490d3e
Nov 03 00:20:11.092721 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:11.092690    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9173f32c-3ccb-4034-91d4-326e14625e94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4da3e6a9-dgsvs\" (UID: \"9173f32c-3ccb-4034-91d4-326e14625e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da3e6a9-dgsvs"
Nov 03 00:20:11.092922 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:11.092887    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9173f32c-3ccb-4034-91d4-326e14625e94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4da3e6a9-dgsvs\" (UID: \"9173f32c-3ccb-4034-91d4-326e14625e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da3e6a9-dgsvs"
Nov 03 00:20:11.092999 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:11.092980    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsx9v\" (UniqueName: \"kubernetes.io/projected/9173f32c-3ccb-4034-91d4-326e14625e94-kube-api-access-wsx9v\") pod \"maas-api-revoke-sub-4da3e6a9-dgsvs\" (UID: \"9173f32c-3ccb-4034-91d4-326e14625e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da3e6a9-dgsvs"
Nov 03 00:20:11.193740 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:11.193647    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsx9v\" (UniqueName: \"kubernetes.io/projected/9173f32c-3ccb-4034-91d4-326e14625e94-kube-api-access-wsx9v\") pod \"maas-api-revoke-sub-4da3e6a9-dgsvs\" (UID: \"9173f32c-3ccb-4034-91d4-326e14625e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da3e6a9-dgsvs"
Nov 03 00:20:11.193963 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:11.193770    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9173f32c-3ccb-4034-91d4-326e14625e94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4da3e6a9-dgsvs\" (UID: \"9173f32c-3ccb-4034-91d4-326e14625e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da3e6a9-dgsvs"
Nov 03 00:20:11.193963 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:11.193882    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9173f32c-3ccb-4034-91d4-326e14625e94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4da3e6a9-dgsvs\" (UID: \"9173f32c-3ccb-4034-91d4-326e14625e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da3e6a9-dgsvs"
Nov 03 00:20:11.194720 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:11.194689    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9173f32c-3ccb-4034-91d4-326e14625e94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4da3e6a9-dgsvs\" (UID: \"9173f32c-3ccb-4034-91d4-326e14625e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da3e6a9-dgsvs"
Nov 03 00:20:11.196310 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:11.196280    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9173f32c-3ccb-4034-91d4-326e14625e94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4da3e6a9-dgsvs\" (UID: \"9173f32c-3ccb-4034-91d4-326e14625e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da3e6a9-dgsvs"
Nov 03 00:20:11.210977 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:11.210942    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsx9v\" (UniqueName: \"kubernetes.io/projected/9173f32c-3ccb-4034-91d4-326e14625e94-kube-api-access-wsx9v\") pod \"maas-api-revoke-sub-4da3e6a9-dgsvs\" (UID: \"9173f32c-3ccb-4034-91d4-326e14625e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da3e6a9-dgsvs"
Nov 03 00:20:11.251902 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:11.251872    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da3e6a9-dgsvs"
Nov 03 00:20:11.398739 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:11.398689    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p" event={"ID":"c36d8715-10c0-4e43-9ae1-e33445533437","Type":"ContainerStarted","Data":"5624f1db9e7cca13741f4cd014f27732798e82e61c2810bde4a7a0d1efebedf6"}
Nov 03 00:20:11.398970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:11.398772    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p" event={"ID":"c36d8715-10c0-4e43-9ae1-e33445533437","Type":"ContainerStarted","Data":"14d2f69ccf88fb864412a351a1e91e6ce05b679ddffab565518e6afef4490d3e"}
Nov 03 00:20:11.399806 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:11.399778    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p"
Nov 03 00:20:11.436683 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:11.436623    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p" podStartSLOduration=1.436608366 podStartE2EDuration="1.436608366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:11.432845877 +0000 UTC m=+894.312429920" watchObservedRunningTime="2026-11-03 00:20:11.436608366 +0000 UTC m=+894.316192409"
Nov 03 00:20:11.556035 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:11.555998    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4da3e6a9-dgsvs"]
Nov 03 00:20:11.556451 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:20:11.556417    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9173f32c_3ccb_4034_91d4_326e14625e94.slice/crio-006265a7e82b3d4832940ec48439c4ceef2647c608e4630b1539e9b08a785f5b WatchSource:0}: Error finding container 006265a7e82b3d4832940ec48439c4ceef2647c608e4630b1539e9b08a785f5b: Status 404 returned error can't find the container with id 006265a7e82b3d4832940ec48439c4ceef2647c608e4630b1539e9b08a785f5b
Nov 03 00:20:12.297637 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:12.297550    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ef73dc5-mlftf"]
Nov 03 00:20:12.303314 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:12.303280    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef73dc5-mlftf"
Nov 03 00:20:12.324644 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:12.324310    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ef73dc5-mlftf"]
Nov 03 00:20:12.417577 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:12.417528    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxpmf\" (UniqueName: \"kubernetes.io/projected/9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231-kube-api-access-mxpmf\") pod \"maas-api-revoke-sub-0ef73dc5-mlftf\" (UID: \"9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef73dc5-mlftf"
Nov 03 00:20:12.417781 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:12.417746    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ef73dc5-mlftf\" (UID: \"9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef73dc5-mlftf"
Nov 03 00:20:12.417902 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:12.417882    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ef73dc5-mlftf\" (UID: \"9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef73dc5-mlftf"
Nov 03 00:20:12.418247 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:12.418220    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da3e6a9-dgsvs" event={"ID":"9173f32c-3ccb-4034-91d4-326e14625e94","Type":"ContainerStarted","Data":"17b01dfebc2fd51d74800a20fa37f2e2a27632804126da7fae17dbf2c46eb40c"}
Nov 03 00:20:12.418328 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:12.418282    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da3e6a9-dgsvs" event={"ID":"9173f32c-3ccb-4034-91d4-326e14625e94","Type":"ContainerStarted","Data":"006265a7e82b3d4832940ec48439c4ceef2647c608e4630b1539e9b08a785f5b"}
Nov 03 00:20:12.456196 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:12.456135    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da3e6a9-dgsvs" podStartSLOduration=2.456119804 podStartE2EDuration="2.456119804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:12.452193588 +0000 UTC m=+895.331777628" watchObservedRunningTime="2026-11-03 00:20:12.456119804 +0000 UTC m=+895.335703854"
Nov 03 00:20:12.469121 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:12.469070    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:20:12.469303 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:12.469139    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5" podUID="ba16a59b-3dc5-4c8e-8378-70b27bdd0590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:12.519339 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:12.519287    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ef73dc5-mlftf\" (UID: \"9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef73dc5-mlftf"
Nov 03 00:20:12.519561 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:12.519457    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ef73dc5-mlftf\" (UID: \"9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef73dc5-mlftf"
Nov 03 00:20:12.519638 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:12.519590    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxpmf\" (UniqueName: \"kubernetes.io/projected/9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231-kube-api-access-mxpmf\") pod \"maas-api-revoke-sub-0ef73dc5-mlftf\" (UID: \"9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef73dc5-mlftf"
Nov 03 00:20:12.520701 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:12.520671    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ef73dc5-mlftf\" (UID: \"9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef73dc5-mlftf"
Nov 03 00:20:12.523056 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:12.523029    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ef73dc5-mlftf\" (UID: \"9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef73dc5-mlftf"
Nov 03 00:20:12.537223 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:12.537177    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxpmf\" (UniqueName: \"kubernetes.io/projected/9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231-kube-api-access-mxpmf\") pod \"maas-api-revoke-sub-0ef73dc5-mlftf\" (UID: \"9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef73dc5-mlftf"
Nov 03 00:20:12.642387 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:12.642348    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef73dc5-mlftf"
Nov 03 00:20:12.828832 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:12.828795    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:12.829460 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:12.829383    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:12.831194 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:12.831142    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:20:12.953014 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:12.952983    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ef73dc5-mlftf"]
Nov 03 00:20:12.954088 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:20:12.954056    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9638d4f5_5ac7_48b0_a2e6_2eed8fa2f231.slice/crio-43f17ceea0bc0846c948dc0d1eb9bc7287a629b00f79687862da3a3c666e980b WatchSource:0}: Error finding container 43f17ceea0bc0846c948dc0d1eb9bc7287a629b00f79687862da3a3c666e980b: Status 404 returned error can't find the container with id 43f17ceea0bc0846c948dc0d1eb9bc7287a629b00f79687862da3a3c666e980b
Nov 03 00:20:13.432741 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:13.432699    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef73dc5-mlftf" event={"ID":"9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231","Type":"ContainerStarted","Data":"112c8521e34643047e0d702ed07444dd457f4a1dbd4671fa3c27d0e9515a8403"}
Nov 03 00:20:13.432976 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:13.432762    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef73dc5-mlftf" event={"ID":"9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231","Type":"ContainerStarted","Data":"43f17ceea0bc0846c948dc0d1eb9bc7287a629b00f79687862da3a3c666e980b"}
Nov 03 00:20:13.434495 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:13.434467    5012 generic.go:356] "Generic (PLEG): container finished" podID="9173f32c-3ccb-4034-91d4-326e14625e94" containerID="17b01dfebc2fd51d74800a20fa37f2e2a27632804126da7fae17dbf2c46eb40c" exitCode=0
Nov 03 00:20:13.434638 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:13.434556    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da3e6a9-dgsvs" event={"ID":"9173f32c-3ccb-4034-91d4-326e14625e94","Type":"ContainerDied","Data":"17b01dfebc2fd51d74800a20fa37f2e2a27632804126da7fae17dbf2c46eb40c"}
Nov 03 00:20:13.493622 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:13.493515    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef73dc5-mlftf" podStartSLOduration=1.4934978509999999 podStartE2EDuration="1.493497851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:13.489716807 +0000 UTC m=+896.369300850" watchObservedRunningTime="2026-11-03 00:20:13.493497851 +0000 UTC m=+896.373081951"
Nov 03 00:20:14.444929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:14.444881    5012 generic.go:356] "Generic (PLEG): container finished" podID="9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231" containerID="112c8521e34643047e0d702ed07444dd457f4a1dbd4671fa3c27d0e9515a8403" exitCode=0
Nov 03 00:20:14.445140 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:14.444960    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef73dc5-mlftf" event={"ID":"9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231","Type":"ContainerDied","Data":"112c8521e34643047e0d702ed07444dd457f4a1dbd4671fa3c27d0e9515a8403"}
Nov 03 00:20:14.632278 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:14.632240    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da3e6a9-dgsvs"
Nov 03 00:20:14.749061 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:14.748975    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9173f32c-3ccb-4034-91d4-326e14625e94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9173f32c-3ccb-4034-91d4-326e14625e94-maas-api-xks-ca\") pod \"9173f32c-3ccb-4034-91d4-326e14625e94\" (UID: \"9173f32c-3ccb-4034-91d4-326e14625e94\") "
Nov 03 00:20:14.749220 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:14.749062    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9173f32c-3ccb-4034-91d4-326e14625e94-kube-api-access-wsx9v\" (UniqueName: \"kubernetes.io/projected/9173f32c-3ccb-4034-91d4-326e14625e94-kube-api-access-wsx9v\") pod \"9173f32c-3ccb-4034-91d4-326e14625e94\" (UID: \"9173f32c-3ccb-4034-91d4-326e14625e94\") "
Nov 03 00:20:14.749220 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:14.749105    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9173f32c-3ccb-4034-91d4-326e14625e94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9173f32c-3ccb-4034-91d4-326e14625e94-maas-api-openshift-ca\") pod \"9173f32c-3ccb-4034-91d4-326e14625e94\" (UID: \"9173f32c-3ccb-4034-91d4-326e14625e94\") "
Nov 03 00:20:14.749495 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:14.749465    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9173f32c-3ccb-4034-91d4-326e14625e94-maas-api-openshift-ca" pod "9173f32c-3ccb-4034-91d4-326e14625e94" (UID: "9173f32c-3ccb-4034-91d4-326e14625e94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:20:14.749681 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:14.749662    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9173f32c-3ccb-4034-91d4-326e14625e94-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:14.751220 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:14.751196    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9173f32c-3ccb-4034-91d4-326e14625e94-maas-api-xks-ca" pod "9173f32c-3ccb-4034-91d4-326e14625e94" (UID: "9173f32c-3ccb-4034-91d4-326e14625e94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:20:14.751220 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:14.751204    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9173f32c-3ccb-4034-91d4-326e14625e94-kube-api-access-wsx9v" pod "9173f32c-3ccb-4034-91d4-326e14625e94" (UID: "9173f32c-3ccb-4034-91d4-326e14625e94"). InnerVolumeSpecName "kube-api-access-wsx9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:20:14.850434 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:14.850399    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9173f32c-3ccb-4034-91d4-326e14625e94-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:14.850434 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:14.850429    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsx9v\" (UniqueName: \"kubernetes.io/projected/9173f32c-3ccb-4034-91d4-326e14625e94-kube-api-access-wsx9v\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:15.456436 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:15.456385    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da3e6a9-dgsvs"
Nov 03 00:20:15.457290 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:15.457253    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da3e6a9-dgsvs" event={"ID":"9173f32c-3ccb-4034-91d4-326e14625e94","Type":"ContainerDied","Data":"006265a7e82b3d4832940ec48439c4ceef2647c608e4630b1539e9b08a785f5b"}
Nov 03 00:20:15.457403 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:15.457353    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="006265a7e82b3d4832940ec48439c4ceef2647c608e4630b1539e9b08a785f5b"
Nov 03 00:20:15.602407 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:15.602380    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef73dc5-mlftf"
Nov 03 00:20:15.760746 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:15.760704    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231-kube-api-access-mxpmf\" (UniqueName: \"kubernetes.io/projected/9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231-kube-api-access-mxpmf\") pod \"9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231\" (UID: \"9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231\") "
Nov 03 00:20:15.761153 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:15.760779    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231-maas-api-xks-ca\") pod \"9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231\" (UID: \"9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231\") "
Nov 03 00:20:15.761153 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:15.760823    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231-maas-api-openshift-ca\") pod \"9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231\" (UID: \"9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231\") "
Nov 03 00:20:15.761228 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:15.761177    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231-maas-api-openshift-ca" pod "9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231" (UID: "9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:20:15.762789 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:15.762770    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231-maas-api-xks-ca" pod "9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231" (UID: "9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:20:15.762937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:15.762904    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231-kube-api-access-mxpmf" pod "9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231" (UID: "9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231"). InnerVolumeSpecName "kube-api-access-mxpmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:20:15.862346 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:15.862321    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:15.862346 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:15.862347    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:15.862525 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:15.862361    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxpmf\" (UniqueName: \"kubernetes.io/projected/9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231-kube-api-access-mxpmf\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:16.464806 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:16.464775    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef73dc5-mlftf"
Nov 03 00:20:16.465021 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:16.464772    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ef73dc5-mlftf" event={"ID":"9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231","Type":"ContainerDied","Data":"43f17ceea0bc0846c948dc0d1eb9bc7287a629b00f79687862da3a3c666e980b"}
Nov 03 00:20:16.465021 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:16.464929    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43f17ceea0bc0846c948dc0d1eb9bc7287a629b00f79687862da3a3c666e980b"
Nov 03 00:20:17.417621 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:17.417581    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 00:20:17.419465 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:17.419437    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 00:20:17.422506 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:17.422480    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:20:17.424083 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:17.424050    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:20:18.222091 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.222059    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"]
Nov 03 00:20:18.222693 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.222677    5012 cpu_manager.go:414] "containerMap: removing container" podUID="9173f32c-3ccb-4034-91d4-326e14625e94" containerName="revoke-keys"
Nov 03 00:20:18.222747 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.222695    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9173f32c-3ccb-4034-91d4-326e14625e94" containerName="revoke-keys"
Nov 03 00:20:18.222747 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.222722    5012 cpu_manager.go:414] "containerMap: removing container" podUID="9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231" containerName="revoke-keys"
Nov 03 00:20:18.222747 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.222727    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231" containerName="revoke-keys"
Nov 03 00:20:18.222841 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.222826    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="9173f32c-3ccb-4034-91d4-326e14625e94" containerName="revoke-keys"
Nov 03 00:20:18.222841 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.222836    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231" containerName="revoke-keys"
Nov 03 00:20:18.228945 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.228126    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.255022 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.254986    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"]
Nov 03 00:20:18.388931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.388870    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.388931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.388933    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.389209 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.389195    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b0a0f0-c396-4b0d-8f77-f032a1174374-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.389279 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.389233    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.389279 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.389248    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4krc\" (UniqueName: \"kubernetes.io/projected/34b0a0f0-c396-4b0d-8f77-f032a1174374-kube-api-access-p4krc\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.389383 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.389316    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.389383 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.389361    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.389588 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.389569    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.389666 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.389642    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.490988 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.490890    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.490988 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.490979    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.491380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.491020    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.491380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.491047    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.491380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.491148    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b0a0f0-c396-4b0d-8f77-f032a1174374-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.491380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.491193    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.491380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.491219    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4krc\" (UniqueName: \"kubernetes.io/projected/34b0a0f0-c396-4b0d-8f77-f032a1174374-kube-api-access-p4krc\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.491380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.491249    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.491380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.491286    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.491595 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.491409    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.491775 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.491250    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.491775 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.491765    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.492053 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.492023    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.492477 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.492457    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b0a0f0-c396-4b0d-8f77-f032a1174374-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.493927 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.493882    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.494022 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.493947    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.507964 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.507937    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.508098 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.508083    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4krc\" (UniqueName: \"kubernetes.io/projected/34b0a0f0-c396-4b0d-8f77-f032a1174374-kube-api-access-p4krc\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.572276 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.572242    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:18.905743 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.905706    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"]
Nov 03 00:20:18.908589 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.908535    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:18.908759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.908635    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:18.908759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:18.908684    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:19.537845 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:19.537799    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl" event={"ID":"34b0a0f0-c396-4b0d-8f77-f032a1174374","Type":"ContainerStarted","Data":"ef14ea28f3fa6621aebf8876a281d867ece48136afdb411cc6003eb9d71e5aaf"}
Nov 03 00:20:19.556074 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:19.556016    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl" event={"ID":"34b0a0f0-c396-4b0d-8f77-f032a1174374","Type":"ContainerStarted","Data":"232098ea91feae1fd1ce51dd174eb72f493d9a2c144f3090c1e73b364d136701"}
Nov 03 00:20:19.574630 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:19.574598    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:19.576983 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:19.576945    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 00:20:19.577273 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:19.577241    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl" podUID="34b0a0f0-c396-4b0d-8f77-f032a1174374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 00:20:19.650188 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:19.650125    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl" podStartSLOduration=1.6501067759999999 podStartE2EDuration="1.650106776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:19.624520665 +0000 UTC m=+902.504104729" watchObservedRunningTime="2026-11-03 00:20:19.650106776 +0000 UTC m=+902.529690820"
Nov 03 00:20:20.576934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:20.576651    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 00:20:20.576934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:20.576718    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl" podUID="34b0a0f0-c396-4b0d-8f77-f032a1174374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 00:20:20.972105 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:20.972033    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:21.093092 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:21.093028    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:21.585475 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:21.585420    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 00:20:21.589936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:21.586090    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl" podUID="34b0a0f0-c396-4b0d-8f77-f032a1174374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 00:20:21.976068 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:21.976009    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 03 00:20:21.977057 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:21.977028    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:20:22.007947 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:22.007874    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:22.382015 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:22.381006    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-5lnf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:20:22.382015 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:22.381087    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2" podUID="aa430510-e80c-4743-b4f3-838ce423e442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:22.448942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:22.448627    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:22.456943 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:22.449456    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:20:22.608105 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:22.608056    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 00:20:22.608630 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:22.608132    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl" podUID="34b0a0f0-c396-4b0d-8f77-f032a1174374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 00:20:22.639606 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:22.639150    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:20:23.011110 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:23.011006    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Nov 03 00:20:23.011731 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:23.011706    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:20:23.032641 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:23.032589    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:23.567956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:23.548126    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p"
Nov 03 00:20:23.570430 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:23.568346    5012 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p" secret="" err="secret \"payload-processing-external-model-dockercfg-xtc2k\" not found"
Nov 03 00:20:23.585594 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:23.585536    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 00:20:23.586000 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:23.585959    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl" podUID="34b0a0f0-c396-4b0d-8f77-f032a1174374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 00:20:23.622019 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:23.621970    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p"]
Nov 03 00:20:23.635359 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:23.635306    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:23.673936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:23.673206    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8"]
Nov 03 00:20:23.799189 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:20:23.797578    5012 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b39c24993f8f: secret "agc-cred-b39c24993f8f" not found
Nov 03 00:20:23.799189 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:20:23.797614    5012 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p: secret "agc-cred-b39c24993f8f" not found
Nov 03 00:20:23.799189 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:20:23.797695    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c36d8715-10c0-4e43-9ae1-e33445533437-provider-credentials podName:c36d8715-10c0-4e43-9ae1-e33445533437 nodeName:}" failed. No retries permitted until 2026-11-03 00:20:24.297668856 +0000 UTC m=+907.177252893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c36d8715-10c0-4e43-9ae1-e33445533437-provider-credentials") pod "payload-processing-external-model-6d94cc47fc-4tc2p" (UID: "c36d8715-10c0-4e43-9ae1-e33445533437") : secret "agc-cred-b39c24993f8f" not found
Nov 03 00:20:23.799189 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:20:23.798285    5012 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Nov 03 00:20:23.799189 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:20:23.798349    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c36d8715-10c0-4e43-9ae1-e33445533437-plugins-config-volume podName:c36d8715-10c0-4e43-9ae1-e33445533437 nodeName:}" failed. No retries permitted until 2026-11-03 00:20:24.298329464 +0000 UTC m=+907.177913499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/c36d8715-10c0-4e43-9ae1-e33445533437-plugins-config-volume") pod "payload-processing-external-model-6d94cc47fc-4tc2p" (UID: "c36d8715-10c0-4e43-9ae1-e33445533437") : configmap "payload-processing-external-model-plugins" not found
Nov 03 00:20:23.803095 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:23.803034    5012 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:24.042941 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:24.039372    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:24.097933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:24.093115    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:24.307997 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:20:24.307379    5012 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Nov 03 00:20:24.307997 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:20:24.307468    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c36d8715-10c0-4e43-9ae1-e33445533437-plugins-config-volume podName:c36d8715-10c0-4e43-9ae1-e33445533437 nodeName:}" failed. No retries permitted until 2026-11-03 00:20:25.30744795 +0000 UTC m=+908.187031990 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/c36d8715-10c0-4e43-9ae1-e33445533437-plugins-config-volume") pod "payload-processing-external-model-6d94cc47fc-4tc2p" (UID: "c36d8715-10c0-4e43-9ae1-e33445533437") : configmap "payload-processing-external-model-plugins" not found
Nov 03 00:20:24.308269 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:20:24.308070    5012 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b39c24993f8f: secret "agc-cred-b39c24993f8f" not found
Nov 03 00:20:24.308269 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:20:24.308094    5012 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p: secret "agc-cred-b39c24993f8f" not found
Nov 03 00:20:24.308269 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:20:24.308147    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c36d8715-10c0-4e43-9ae1-e33445533437-provider-credentials podName:c36d8715-10c0-4e43-9ae1-e33445533437 nodeName:}" failed. No retries permitted until 2026-11-03 00:20:25.308129707 +0000 UTC m=+908.187713745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c36d8715-10c0-4e43-9ae1-e33445533437-provider-credentials") pod "payload-processing-external-model-6d94cc47fc-4tc2p" (UID: "c36d8715-10c0-4e43-9ae1-e33445533437") : secret "agc-cred-b39c24993f8f" not found
Nov 03 00:20:24.596011 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:24.594392    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 00:20:24.596011 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:24.594473    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl" podUID="34b0a0f0-c396-4b0d-8f77-f032a1174374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 00:20:24.657937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:24.636189    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:24.657937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:24.637032    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:20:24.860980 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:24.860059    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:20:24.863766 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:24.863721    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:20:24.884034 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:24.884002    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:20:25.362154 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:20:25.362122    5012 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b39c24993f8f: secret "agc-cred-b39c24993f8f" not found
Nov 03 00:20:25.362154 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:20:25.362152    5012 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p: secret "agc-cred-b39c24993f8f" not found
Nov 03 00:20:25.362427 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:20:25.362195    5012 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Nov 03 00:20:25.362427 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:20:25.362227    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c36d8715-10c0-4e43-9ae1-e33445533437-provider-credentials podName:c36d8715-10c0-4e43-9ae1-e33445533437 nodeName:}" failed. No retries permitted until 2026-11-03 00:20:27.362204199 +0000 UTC m=+910.241788239 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c36d8715-10c0-4e43-9ae1-e33445533437-provider-credentials") pod "payload-processing-external-model-6d94cc47fc-4tc2p" (UID: "c36d8715-10c0-4e43-9ae1-e33445533437") : secret "agc-cred-b39c24993f8f" not found
Nov 03 00:20:25.362427 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:20:25.362264    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c36d8715-10c0-4e43-9ae1-e33445533437-plugins-config-volume podName:c36d8715-10c0-4e43-9ae1-e33445533437 nodeName:}" failed. No retries permitted until 2026-11-03 00:20:27.362244944 +0000 UTC m=+910.241828968 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/c36d8715-10c0-4e43-9ae1-e33445533437-plugins-config-volume") pod "payload-processing-external-model-6d94cc47fc-4tc2p" (UID: "c36d8715-10c0-4e43-9ae1-e33445533437") : configmap "payload-processing-external-model-plugins" not found
Nov 03 00:20:25.573241 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:25.573210    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 00:20:25.573427 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:25.573262    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl" podUID="34b0a0f0-c396-4b0d-8f77-f032a1174374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 00:20:25.728265 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:25.728181    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fe2ca7-fj2rz"]
Nov 03 00:20:25.732336 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:25.732319    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fe2ca7-fj2rz"
Nov 03 00:20:25.746682 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:25.746652    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:20:25.746682 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:25.746664    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 00:20:25.746834 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:25.746682    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:20:25.751207 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:25.751185    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fe2ca7-fj2rz"]
Nov 03 00:20:25.756061 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:25.756038    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 00:20:25.870302 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:25.870264    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7121c2a-6dcf-4671-855b-07bafc66e27d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06fe2ca7-fj2rz\" (UID: \"d7121c2a-6dcf-4671-855b-07bafc66e27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fe2ca7-fj2rz"
Nov 03 00:20:25.870302 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:25.870308    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7121c2a-6dcf-4671-855b-07bafc66e27d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06fe2ca7-fj2rz\" (UID: \"d7121c2a-6dcf-4671-855b-07bafc66e27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fe2ca7-fj2rz"
Nov 03 00:20:25.870533 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:25.870479    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9zlp\" (UniqueName: \"kubernetes.io/projected/d7121c2a-6dcf-4671-855b-07bafc66e27d-kube-api-access-h9zlp\") pod \"maas-api-revoke-sub-06fe2ca7-fj2rz\" (UID: \"d7121c2a-6dcf-4671-855b-07bafc66e27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fe2ca7-fj2rz"
Nov 03 00:20:25.972403 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:25.972363    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7121c2a-6dcf-4671-855b-07bafc66e27d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06fe2ca7-fj2rz\" (UID: \"d7121c2a-6dcf-4671-855b-07bafc66e27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fe2ca7-fj2rz"
Nov 03 00:20:25.972613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:25.972421    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7121c2a-6dcf-4671-855b-07bafc66e27d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06fe2ca7-fj2rz\" (UID: \"d7121c2a-6dcf-4671-855b-07bafc66e27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fe2ca7-fj2rz"
Nov 03 00:20:25.972613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:25.972556    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9zlp\" (UniqueName: \"kubernetes.io/projected/d7121c2a-6dcf-4671-855b-07bafc66e27d-kube-api-access-h9zlp\") pod \"maas-api-revoke-sub-06fe2ca7-fj2rz\" (UID: \"d7121c2a-6dcf-4671-855b-07bafc66e27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fe2ca7-fj2rz"
Nov 03 00:20:25.973566 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:25.973533    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7121c2a-6dcf-4671-855b-07bafc66e27d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06fe2ca7-fj2rz\" (UID: \"d7121c2a-6dcf-4671-855b-07bafc66e27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fe2ca7-fj2rz"
Nov 03 00:20:25.976066 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:25.976040    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7121c2a-6dcf-4671-855b-07bafc66e27d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06fe2ca7-fj2rz\" (UID: \"d7121c2a-6dcf-4671-855b-07bafc66e27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fe2ca7-fj2rz"
Nov 03 00:20:26.013981 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:26.004700    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9zlp\" (UniqueName: \"kubernetes.io/projected/d7121c2a-6dcf-4671-855b-07bafc66e27d-kube-api-access-h9zlp\") pod \"maas-api-revoke-sub-06fe2ca7-fj2rz\" (UID: \"d7121c2a-6dcf-4671-855b-07bafc66e27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fe2ca7-fj2rz"
Nov 03 00:20:26.047633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:26.047598    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fe2ca7-fj2rz"
Nov 03 00:20:26.371973 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:20:26.371902    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7121c2a_6dcf_4671_855b_07bafc66e27d.slice/crio-d6d1fe78629ade569efe9ef3a0cc861a33d62ff564c7b33fa2a580790b86996e WatchSource:0}: Error finding container d6d1fe78629ade569efe9ef3a0cc861a33d62ff564c7b33fa2a580790b86996e: Status 404 returned error can't find the container with id d6d1fe78629ade569efe9ef3a0cc861a33d62ff564c7b33fa2a580790b86996e
Nov 03 00:20:26.372242 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:26.372219    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fe2ca7-fj2rz"]
Nov 03 00:20:26.573022 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:26.572992    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 00:20:26.573188 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:26.573042    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl" podUID="34b0a0f0-c396-4b0d-8f77-f032a1174374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 00:20:27.080674 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:27.080570    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fe2ca7-fj2rz" event={"ID":"d7121c2a-6dcf-4671-855b-07bafc66e27d","Type":"ContainerStarted","Data":"882061f0ed159577ec5b662fcd0e074639efab3bc4e850461f38285a1f0ace66"}
Nov 03 00:20:27.080674 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:27.080634    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fe2ca7-fj2rz" event={"ID":"d7121c2a-6dcf-4671-855b-07bafc66e27d","Type":"ContainerStarted","Data":"d6d1fe78629ade569efe9ef3a0cc861a33d62ff564c7b33fa2a580790b86996e"}
Nov 03 00:20:27.121994 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:27.121899    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fe2ca7-fj2rz" podStartSLOduration=2.121884783 podStartE2EDuration="2.121884783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:27.12144136 +0000 UTC m=+910.001025405" watchObservedRunningTime="2026-11-03 00:20:27.121884783 +0000 UTC m=+910.001468825"
Nov 03 00:20:27.412989 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:20:27.412215    5012 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b39c24993f8f: secret "agc-cred-b39c24993f8f" not found
Nov 03 00:20:27.412989 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:20:27.412254    5012 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p: secret "agc-cred-b39c24993f8f" not found
Nov 03 00:20:27.412989 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:20:27.412314    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c36d8715-10c0-4e43-9ae1-e33445533437-provider-credentials podName:c36d8715-10c0-4e43-9ae1-e33445533437 nodeName:}" failed. No retries permitted until 2026-11-03 00:20:31.412294936 +0000 UTC m=+914.291878972 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c36d8715-10c0-4e43-9ae1-e33445533437-provider-credentials") pod "payload-processing-external-model-6d94cc47fc-4tc2p" (UID: "c36d8715-10c0-4e43-9ae1-e33445533437") : secret "agc-cred-b39c24993f8f" not found
Nov 03 00:20:27.413628 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:20:27.413175    5012 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Nov 03 00:20:27.413628 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:20:27.413228    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c36d8715-10c0-4e43-9ae1-e33445533437-plugins-config-volume podName:c36d8715-10c0-4e43-9ae1-e33445533437 nodeName:}" failed. No retries permitted until 2026-11-03 00:20:31.413213078 +0000 UTC m=+914.292797104 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/c36d8715-10c0-4e43-9ae1-e33445533437-plugins-config-volume") pod "payload-processing-external-model-6d94cc47fc-4tc2p" (UID: "c36d8715-10c0-4e43-9ae1-e33445533437") : configmap "payload-processing-external-model-plugins" not found
Nov 03 00:20:27.573057 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:27.573019    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 00:20:27.573230 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:27.573080    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl" podUID="34b0a0f0-c396-4b0d-8f77-f032a1174374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 00:20:28.113166 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:28.113126    5012 generic.go:356] "Generic (PLEG): container finished" podID="d7121c2a-6dcf-4671-855b-07bafc66e27d" containerID="882061f0ed159577ec5b662fcd0e074639efab3bc4e850461f38285a1f0ace66" exitCode=0
Nov 03 00:20:28.113607 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:28.113216    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fe2ca7-fj2rz" event={"ID":"d7121c2a-6dcf-4671-855b-07bafc66e27d","Type":"ContainerDied","Data":"882061f0ed159577ec5b662fcd0e074639efab3bc4e850461f38285a1f0ace66"}
Nov 03 00:20:28.572721 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:28.572639    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 00:20:28.572721 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:28.572696    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl" podUID="34b0a0f0-c396-4b0d-8f77-f032a1174374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 00:20:28.674061 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:28.674017    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8" podUID="73e79035-3352-4932-801f-db7ced694849" containerName="payload-processing" containerID="cri-o://3e4512568950fe1b157ee2f35a69d793d2ff8449276279ddd617528c16ed2023" gracePeriod=25
Nov 03 00:20:28.972815 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:28.972786    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8"
Nov 03 00:20:29.043832 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.043786    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p" podUID="c36d8715-10c0-4e43-9ae1-e33445533437" containerName="payload-processing" containerID="cri-o://5624f1db9e7cca13741f4cd014f27732798e82e61c2810bde4a7a0d1efebedf6" gracePeriod=25
Nov 03 00:20:29.125192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.125154    5012 generic.go:356] "Generic (PLEG): container finished" podID="73e79035-3352-4932-801f-db7ced694849" containerID="3e4512568950fe1b157ee2f35a69d793d2ff8449276279ddd617528c16ed2023" exitCode=0
Nov 03 00:20:29.125607 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.125226    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8"
Nov 03 00:20:29.125607 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.125236    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8" event={"ID":"73e79035-3352-4932-801f-db7ced694849","Type":"ContainerDied","Data":"3e4512568950fe1b157ee2f35a69d793d2ff8449276279ddd617528c16ed2023"}
Nov 03 00:20:29.125607 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.125321    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8" event={"ID":"73e79035-3352-4932-801f-db7ced694849","Type":"ContainerDied","Data":"d6aa316a4aafbdca43864587e84024b43432c2fff5d82d3730fcad7815fc2e61"}
Nov 03 00:20:29.125607 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.125346    5012 scope.go:122] "RemoveContainer" containerID="3e4512568950fe1b157ee2f35a69d793d2ff8449276279ddd617528c16ed2023"
Nov 03 00:20:29.134995 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.134966    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73e79035-3352-4932-801f-db7ced694849-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/73e79035-3352-4932-801f-db7ced694849-routing-overlay-volume\") pod \"73e79035-3352-4932-801f-db7ced694849\" (UID: \"73e79035-3352-4932-801f-db7ced694849\") "
Nov 03 00:20:29.135114 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.135091    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73e79035-3352-4932-801f-db7ced694849-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e79035-3352-4932-801f-db7ced694849-plugins-config-volume\") pod \"73e79035-3352-4932-801f-db7ced694849\" (UID: \"73e79035-3352-4932-801f-db7ced694849\") "
Nov 03 00:20:29.135238 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.135219    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e79035-3352-4932-801f-db7ced694849-kube-api-access-qfnt4\" (UniqueName: \"kubernetes.io/projected/73e79035-3352-4932-801f-db7ced694849-kube-api-access-qfnt4\") pod \"73e79035-3352-4932-801f-db7ced694849\" (UID: \"73e79035-3352-4932-801f-db7ced694849\") "
Nov 03 00:20:29.135300 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.135283    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e79035-3352-4932-801f-db7ced694849-provider-credentials\" (UniqueName: \"kubernetes.io/projected/73e79035-3352-4932-801f-db7ced694849-provider-credentials\") pod \"73e79035-3352-4932-801f-db7ced694849\" (UID: \"73e79035-3352-4932-801f-db7ced694849\") "
Nov 03 00:20:29.135392 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.135363    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e79035-3352-4932-801f-db7ced694849-routing-overlay-volume" pod "73e79035-3352-4932-801f-db7ced694849" (UID: "73e79035-3352-4932-801f-db7ced694849"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:20:29.135536 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.135497    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e79035-3352-4932-801f-db7ced694849-plugins-config-volume" pod "73e79035-3352-4932-801f-db7ced694849" (UID: "73e79035-3352-4932-801f-db7ced694849"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:20:29.136015 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.135989    5012 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/73e79035-3352-4932-801f-db7ced694849-routing-overlay-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:29.136121 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.136018    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e79035-3352-4932-801f-db7ced694849-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:29.137486 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.137453    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e79035-3352-4932-801f-db7ced694849-provider-credentials" pod "73e79035-3352-4932-801f-db7ced694849" (UID: "73e79035-3352-4932-801f-db7ced694849"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:20:29.137585 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.137509    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e79035-3352-4932-801f-db7ced694849-kube-api-access-qfnt4" pod "73e79035-3352-4932-801f-db7ced694849" (UID: "73e79035-3352-4932-801f-db7ced694849"). InnerVolumeSpecName "kube-api-access-qfnt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:20:29.159737 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.159703    5012 scope.go:122] "RemoveContainer" containerID="3e4512568950fe1b157ee2f35a69d793d2ff8449276279ddd617528c16ed2023"
Nov 03 00:20:29.160161 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:20:29.160131    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e4512568950fe1b157ee2f35a69d793d2ff8449276279ddd617528c16ed2023\": container with ID starting with 3e4512568950fe1b157ee2f35a69d793d2ff8449276279ddd617528c16ed2023 not found: ID does not exist" containerID="3e4512568950fe1b157ee2f35a69d793d2ff8449276279ddd617528c16ed2023"
Nov 03 00:20:29.160265 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.160174    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4512568950fe1b157ee2f35a69d793d2ff8449276279ddd617528c16ed2023"} err="failed to get container status \"3e4512568950fe1b157ee2f35a69d793d2ff8449276279ddd617528c16ed2023\": rpc error: code = NotFound desc = could not find container \"3e4512568950fe1b157ee2f35a69d793d2ff8449276279ddd617528c16ed2023\": container with ID starting with 3e4512568950fe1b157ee2f35a69d793d2ff8449276279ddd617528c16ed2023 not found: ID does not exist"
Nov 03 00:20:29.238890 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.238816    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfnt4\" (UniqueName: \"kubernetes.io/projected/73e79035-3352-4932-801f-db7ced694849-kube-api-access-qfnt4\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:29.238890 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.238853    5012 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/73e79035-3352-4932-801f-db7ced694849-provider-credentials\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:29.300998 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.300969    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fe2ca7-fj2rz"
Nov 03 00:20:29.342233 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.342196    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p"
Nov 03 00:20:29.440317 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.440227    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7121c2a-6dcf-4671-855b-07bafc66e27d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7121c2a-6dcf-4671-855b-07bafc66e27d-maas-api-openshift-ca\") pod \"d7121c2a-6dcf-4671-855b-07bafc66e27d\" (UID: \"d7121c2a-6dcf-4671-855b-07bafc66e27d\") "
Nov 03 00:20:29.440492 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.440359    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c36d8715-10c0-4e43-9ae1-e33445533437-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c36d8715-10c0-4e43-9ae1-e33445533437-plugins-config-volume\") pod \"c36d8715-10c0-4e43-9ae1-e33445533437\" (UID: \"c36d8715-10c0-4e43-9ae1-e33445533437\") "
Nov 03 00:20:29.440492 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.440400    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c36d8715-10c0-4e43-9ae1-e33445533437-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c36d8715-10c0-4e43-9ae1-e33445533437-routing-overlay-volume\") pod \"c36d8715-10c0-4e43-9ae1-e33445533437\" (UID: \"c36d8715-10c0-4e43-9ae1-e33445533437\") "
Nov 03 00:20:29.440492 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.440438    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c36d8715-10c0-4e43-9ae1-e33445533437-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c36d8715-10c0-4e43-9ae1-e33445533437-provider-credentials\") pod \"c36d8715-10c0-4e43-9ae1-e33445533437\" (UID: \"c36d8715-10c0-4e43-9ae1-e33445533437\") "
Nov 03 00:20:29.440492 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.440482    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7121c2a-6dcf-4671-855b-07bafc66e27d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7121c2a-6dcf-4671-855b-07bafc66e27d-maas-api-xks-ca\") pod \"d7121c2a-6dcf-4671-855b-07bafc66e27d\" (UID: \"d7121c2a-6dcf-4671-855b-07bafc66e27d\") "
Nov 03 00:20:29.440763 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.440621    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7121c2a-6dcf-4671-855b-07bafc66e27d-kube-api-access-h9zlp\" (UniqueName: \"kubernetes.io/projected/d7121c2a-6dcf-4671-855b-07bafc66e27d-kube-api-access-h9zlp\") pod \"d7121c2a-6dcf-4671-855b-07bafc66e27d\" (UID: \"d7121c2a-6dcf-4671-855b-07bafc66e27d\") "
Nov 03 00:20:29.440763 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.440636    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7121c2a-6dcf-4671-855b-07bafc66e27d-maas-api-openshift-ca" pod "d7121c2a-6dcf-4671-855b-07bafc66e27d" (UID: "d7121c2a-6dcf-4671-855b-07bafc66e27d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:20:29.440763 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.440662    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c36d8715-10c0-4e43-9ae1-e33445533437-kube-api-access-flkz2\" (UniqueName: \"kubernetes.io/projected/c36d8715-10c0-4e43-9ae1-e33445533437-kube-api-access-flkz2\") pod \"c36d8715-10c0-4e43-9ae1-e33445533437\" (UID: \"c36d8715-10c0-4e43-9ae1-e33445533437\") "
Nov 03 00:20:29.440943 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.440783    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c36d8715-10c0-4e43-9ae1-e33445533437-routing-overlay-volume" pod "c36d8715-10c0-4e43-9ae1-e33445533437" (UID: "c36d8715-10c0-4e43-9ae1-e33445533437"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:20:29.441242 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.441218    5012 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c36d8715-10c0-4e43-9ae1-e33445533437-routing-overlay-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:29.441414 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.441397    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7121c2a-6dcf-4671-855b-07bafc66e27d-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:29.441484 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.441415    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c36d8715-10c0-4e43-9ae1-e33445533437-plugins-config-volume" pod "c36d8715-10c0-4e43-9ae1-e33445533437" (UID: "c36d8715-10c0-4e43-9ae1-e33445533437"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:20:29.443342 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.443316    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c36d8715-10c0-4e43-9ae1-e33445533437-kube-api-access-flkz2" pod "c36d8715-10c0-4e43-9ae1-e33445533437" (UID: "c36d8715-10c0-4e43-9ae1-e33445533437"). InnerVolumeSpecName "kube-api-access-flkz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:20:29.443474 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.443349    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7121c2a-6dcf-4671-855b-07bafc66e27d-kube-api-access-h9zlp" pod "d7121c2a-6dcf-4671-855b-07bafc66e27d" (UID: "d7121c2a-6dcf-4671-855b-07bafc66e27d"). InnerVolumeSpecName "kube-api-access-h9zlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:20:29.443474 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.443364    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7121c2a-6dcf-4671-855b-07bafc66e27d-maas-api-xks-ca" pod "d7121c2a-6dcf-4671-855b-07bafc66e27d" (UID: "d7121c2a-6dcf-4671-855b-07bafc66e27d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:20:29.443474 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.443399    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c36d8715-10c0-4e43-9ae1-e33445533437-provider-credentials" pod "c36d8715-10c0-4e43-9ae1-e33445533437" (UID: "c36d8715-10c0-4e43-9ae1-e33445533437"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:20:29.463795 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.463767    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8"]
Nov 03 00:20:29.470387 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.470358    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-f756c6b59-vm8d8"]
Nov 03 00:20:29.543215 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.543175    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c36d8715-10c0-4e43-9ae1-e33445533437-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:29.543215 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.543215    5012 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c36d8715-10c0-4e43-9ae1-e33445533437-provider-credentials\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:29.543408 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.543235    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7121c2a-6dcf-4671-855b-07bafc66e27d-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:29.543408 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.543252    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9zlp\" (UniqueName: \"kubernetes.io/projected/d7121c2a-6dcf-4671-855b-07bafc66e27d-kube-api-access-h9zlp\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:29.543408 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.543271    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flkz2\" (UniqueName: \"kubernetes.io/projected/c36d8715-10c0-4e43-9ae1-e33445533437-kube-api-access-flkz2\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:29.850215 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.850185    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:29.850675 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.850657    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:29.851547 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:29.851526    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:20:30.135524 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:30.135485    5012 generic.go:356] "Generic (PLEG): container finished" podID="c36d8715-10c0-4e43-9ae1-e33445533437" containerID="5624f1db9e7cca13741f4cd014f27732798e82e61c2810bde4a7a0d1efebedf6" exitCode=0
Nov 03 00:20:30.136004 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:30.135532    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p"
Nov 03 00:20:30.136004 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:30.135573    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p" event={"ID":"c36d8715-10c0-4e43-9ae1-e33445533437","Type":"ContainerDied","Data":"5624f1db9e7cca13741f4cd014f27732798e82e61c2810bde4a7a0d1efebedf6"}
Nov 03 00:20:30.136004 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:30.135624    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p" event={"ID":"c36d8715-10c0-4e43-9ae1-e33445533437","Type":"ContainerDied","Data":"14d2f69ccf88fb864412a351a1e91e6ce05b679ddffab565518e6afef4490d3e"}
Nov 03 00:20:30.136004 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:30.135653    5012 scope.go:122] "RemoveContainer" containerID="5624f1db9e7cca13741f4cd014f27732798e82e61c2810bde4a7a0d1efebedf6"
Nov 03 00:20:30.137323 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:30.137309    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fe2ca7-fj2rz"
Nov 03 00:20:30.137416 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:30.137350    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fe2ca7-fj2rz" event={"ID":"d7121c2a-6dcf-4671-855b-07bafc66e27d","Type":"ContainerDied","Data":"d6d1fe78629ade569efe9ef3a0cc861a33d62ff564c7b33fa2a580790b86996e"}
Nov 03 00:20:30.137416 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:30.137398    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6d1fe78629ade569efe9ef3a0cc861a33d62ff564c7b33fa2a580790b86996e"
Nov 03 00:20:30.147181 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:30.147161    5012 scope.go:122] "RemoveContainer" containerID="5624f1db9e7cca13741f4cd014f27732798e82e61c2810bde4a7a0d1efebedf6"
Nov 03 00:20:30.147505 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:20:30.147485    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5624f1db9e7cca13741f4cd014f27732798e82e61c2810bde4a7a0d1efebedf6\": container with ID starting with 5624f1db9e7cca13741f4cd014f27732798e82e61c2810bde4a7a0d1efebedf6 not found: ID does not exist" containerID="5624f1db9e7cca13741f4cd014f27732798e82e61c2810bde4a7a0d1efebedf6"
Nov 03 00:20:30.147553 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:30.147513    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5624f1db9e7cca13741f4cd014f27732798e82e61c2810bde4a7a0d1efebedf6"} err="failed to get container status \"5624f1db9e7cca13741f4cd014f27732798e82e61c2810bde4a7a0d1efebedf6\": rpc error: code = NotFound desc = could not find container \"5624f1db9e7cca13741f4cd014f27732798e82e61c2810bde4a7a0d1efebedf6\": container with ID starting with 5624f1db9e7cca13741f4cd014f27732798e82e61c2810bde4a7a0d1efebedf6 not found: ID does not exist"
Nov 03 00:20:30.196300 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:30.196266    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p"]
Nov 03 00:20:30.207932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:30.201175    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d94cc47fc-4tc2p"]
Nov 03 00:20:31.290709 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:31.290300    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73e79035-3352-4932-801f-db7ced694849" path="/var/lib/kubelet/pods/73e79035-3352-4932-801f-db7ced694849/volumes"
Nov 03 00:20:31.291253 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:31.290997    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c36d8715-10c0-4e43-9ae1-e33445533437" path="/var/lib/kubelet/pods/c36d8715-10c0-4e43-9ae1-e33445533437/volumes"
Nov 03 00:20:33.725935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:33.715761    5012 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": context deadline exceeded"
Nov 03 00:20:33.889810 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:33.889750    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:34.049157 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:34.049040    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:34.769169 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:34.769127    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-8fzrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:20:34.769757 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:34.769721    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc" podUID="51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:34.868039 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:34.867990    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Nov 03 00:20:35.050136 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:35.050025    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 03 00:20:35.050897 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:35.050874    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:20:35.767929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:35.762859    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:20:35.913930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:35.912813    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:37.362484 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:20:37.362444    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:20:38.841726 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:38.841690    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"]
Nov 03 00:20:38.842652 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:38.842627    5012 cpu_manager.go:414] "containerMap: removing container" podUID="73e79035-3352-4932-801f-db7ced694849" containerName="payload-processing"
Nov 03 00:20:38.842731 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:38.842655    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e79035-3352-4932-801f-db7ced694849" containerName="payload-processing"
Nov 03 00:20:38.842731 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:38.842680    5012 cpu_manager.go:414] "containerMap: removing container" podUID="d7121c2a-6dcf-4671-855b-07bafc66e27d" containerName="revoke-keys"
Nov 03 00:20:38.842731 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:38.842688    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7121c2a-6dcf-4671-855b-07bafc66e27d" containerName="revoke-keys"
Nov 03 00:20:38.842731 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:38.842727    5012 cpu_manager.go:414] "containerMap: removing container" podUID="c36d8715-10c0-4e43-9ae1-e33445533437" containerName="payload-processing"
Nov 03 00:20:38.842967 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:38.842736    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c36d8715-10c0-4e43-9ae1-e33445533437" containerName="payload-processing"
Nov 03 00:20:38.842967 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:38.842891    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e79035-3352-4932-801f-db7ced694849" containerName="payload-processing"
Nov 03 00:20:38.842967 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:38.842924    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7121c2a-6dcf-4671-855b-07bafc66e27d" containerName="revoke-keys"
Nov 03 00:20:38.842967 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:38.842942    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="c36d8715-10c0-4e43-9ae1-e33445533437" containerName="payload-processing"
Nov 03 00:20:38.855142 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:38.855119    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:20:38.862454 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:38.862120    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"]
Nov 03 00:20:38.877978 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:38.877952    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8e90c5\"/\"distinct-2-w3-8e90c5-kserve-self-signed-certs\""
Nov 03 00:20:38.955561 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:38.955524    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-model-cache\") pod \"distinct-2-w3-8e90c5-kserve-575dcc5775-l84px\" (UID: \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:20:38.955725 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:38.955583    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-home\") pod \"distinct-2-w3-8e90c5-kserve-575dcc5775-l84px\" (UID: \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:20:38.955725 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:38.955686    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-tmp-dir\") pod \"distinct-2-w3-8e90c5-kserve-575dcc5775-l84px\" (UID: \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:20:38.955725 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:38.955716    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-dshm\") pod \"distinct-2-w3-8e90c5-kserve-575dcc5775-l84px\" (UID: \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:20:38.955835 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:38.955813    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-tls-certs\") pod \"distinct-2-w3-8e90c5-kserve-575dcc5775-l84px\" (UID: \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:20:38.955893 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:38.955878    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjhvq\" (UniqueName: \"kubernetes.io/projected/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-kube-api-access-qjhvq\") pod \"distinct-2-w3-8e90c5-kserve-575dcc5775-l84px\" (UID: \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:20:39.057389 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:39.057332    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-model-cache\") pod \"distinct-2-w3-8e90c5-kserve-575dcc5775-l84px\" (UID: \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:20:39.058813 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:39.058573    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-model-cache\") pod \"distinct-2-w3-8e90c5-kserve-575dcc5775-l84px\" (UID: \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:20:39.059041 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:39.059017    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-home\") pod \"distinct-2-w3-8e90c5-kserve-575dcc5775-l84px\" (UID: \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:20:39.059178 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:39.059025    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-home\") pod \"distinct-2-w3-8e90c5-kserve-575dcc5775-l84px\" (UID: \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:20:39.059381 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:39.059365    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-tmp-dir\") pod \"distinct-2-w3-8e90c5-kserve-575dcc5775-l84px\" (UID: \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:20:39.059518 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:39.059504    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-dshm\") pod \"distinct-2-w3-8e90c5-kserve-575dcc5775-l84px\" (UID: \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:20:39.059723 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:39.059708    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-tls-certs\") pod \"distinct-2-w3-8e90c5-kserve-575dcc5775-l84px\" (UID: \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:20:39.059863 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:39.059847    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjhvq\" (UniqueName: \"kubernetes.io/projected/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-kube-api-access-qjhvq\") pod \"distinct-2-w3-8e90c5-kserve-575dcc5775-l84px\" (UID: \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:20:39.060649 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:39.060630    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-tmp-dir\") pod \"distinct-2-w3-8e90c5-kserve-575dcc5775-l84px\" (UID: \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:20:39.070532 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:39.070503    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-dshm\") pod \"distinct-2-w3-8e90c5-kserve-575dcc5775-l84px\" (UID: \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:20:39.070787 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:39.070756    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-tls-certs\") pod \"distinct-2-w3-8e90c5-kserve-575dcc5775-l84px\" (UID: \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:20:39.078585 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:39.078558    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjhvq\" (UniqueName: \"kubernetes.io/projected/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-kube-api-access-qjhvq\") pod \"distinct-2-w3-8e90c5-kserve-575dcc5775-l84px\" (UID: \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:20:39.171761 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:39.171725    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:20:39.476400 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:20:39.476361    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9b22dc8_f6f1_4c9f_b3ec_28b6ada72095.slice/crio-5620cfb64f1eba25935a4677d0b0314916551cbf39208c1abe493304eaad38e9 WatchSource:0}: Error finding container 5620cfb64f1eba25935a4677d0b0314916551cbf39208c1abe493304eaad38e9: Status 404 returned error can't find the container with id 5620cfb64f1eba25935a4677d0b0314916551cbf39208c1abe493304eaad38e9
Nov 03 00:20:39.476400 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:39.476391    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"]
Nov 03 00:20:40.074660 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:40.074621    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" event={"ID":"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095","Type":"ContainerStarted","Data":"c71c160d763be7b46f3e1c8cda3c3a14c6897d9c0fd32d7f10dbbcf47cc47d20"}
Nov 03 00:20:40.075087 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:40.074674    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" event={"ID":"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095","Type":"ContainerStarted","Data":"5620cfb64f1eba25935a4677d0b0314916551cbf39208c1abe493304eaad38e9"}
Nov 03 00:20:40.084363 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:40.084337    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c387219-4c629"]
Nov 03 00:20:40.088380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:40.088362    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c387219-4c629"
Nov 03 00:20:40.100748 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:40.100607    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:20:40.100748 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:40.100620    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 00:20:40.100748 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:40.100614    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 00:20:40.100748 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:40.100697    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:20:40.104247 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:40.104226    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c387219-4c629"]
Nov 03 00:20:40.115690 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:40.115637    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podStartSLOduration=2.115620235 podStartE2EDuration="2.115620235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:40.110674009 +0000 UTC m=+922.990258055" watchObservedRunningTime="2026-11-03 00:20:40.115620235 +0000 UTC m=+922.995204280"
Nov 03 00:20:40.172793 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:40.172763    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:20:40.173337 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:40.173311    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgs7v\" (UniqueName: \"kubernetes.io/projected/c1fa66d2-1746-4308-bc2a-1f909da2322e-kube-api-access-mgs7v\") pod \"maas-api-revoke-sub-4c387219-4c629\" (UID: \"c1fa66d2-1746-4308-bc2a-1f909da2322e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c387219-4c629"
Nov 03 00:20:40.173391 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:40.173371    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1fa66d2-1746-4308-bc2a-1f909da2322e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c387219-4c629\" (UID: \"c1fa66d2-1746-4308-bc2a-1f909da2322e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c387219-4c629"
Nov 03 00:20:40.173432 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:40.173422    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1fa66d2-1746-4308-bc2a-1f909da2322e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c387219-4c629\" (UID: \"c1fa66d2-1746-4308-bc2a-1f909da2322e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c387219-4c629"
Nov 03 00:20:40.275085 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:40.275047    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgs7v\" (UniqueName: \"kubernetes.io/projected/c1fa66d2-1746-4308-bc2a-1f909da2322e-kube-api-access-mgs7v\") pod \"maas-api-revoke-sub-4c387219-4c629\" (UID: \"c1fa66d2-1746-4308-bc2a-1f909da2322e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c387219-4c629"
Nov 03 00:20:40.275231 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:40.275133    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1fa66d2-1746-4308-bc2a-1f909da2322e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c387219-4c629\" (UID: \"c1fa66d2-1746-4308-bc2a-1f909da2322e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c387219-4c629"
Nov 03 00:20:40.275231 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:40.275217    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1fa66d2-1746-4308-bc2a-1f909da2322e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c387219-4c629\" (UID: \"c1fa66d2-1746-4308-bc2a-1f909da2322e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c387219-4c629"
Nov 03 00:20:40.275995 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:40.275971    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1fa66d2-1746-4308-bc2a-1f909da2322e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c387219-4c629\" (UID: \"c1fa66d2-1746-4308-bc2a-1f909da2322e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c387219-4c629"
Nov 03 00:20:40.277611 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:40.277588    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1fa66d2-1746-4308-bc2a-1f909da2322e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c387219-4c629\" (UID: \"c1fa66d2-1746-4308-bc2a-1f909da2322e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c387219-4c629"
Nov 03 00:20:40.291561 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:40.291531    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgs7v\" (UniqueName: \"kubernetes.io/projected/c1fa66d2-1746-4308-bc2a-1f909da2322e-kube-api-access-mgs7v\") pod \"maas-api-revoke-sub-4c387219-4c629\" (UID: \"c1fa66d2-1746-4308-bc2a-1f909da2322e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c387219-4c629"
Nov 03 00:20:40.402517 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:40.402487    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c387219-4c629"
Nov 03 00:20:40.705723 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:40.705696    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c387219-4c629"]
Nov 03 00:20:40.706323 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:20:40.706297    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1fa66d2_1746_4308_bc2a_1f909da2322e.slice/crio-2ff1b6051160552af377fd0aac5333466be7432fd9c949565bbb5d332a77e240 WatchSource:0}: Error finding container 2ff1b6051160552af377fd0aac5333466be7432fd9c949565bbb5d332a77e240: Status 404 returned error can't find the container with id 2ff1b6051160552af377fd0aac5333466be7432fd9c949565bbb5d332a77e240
Nov 03 00:20:41.092375 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:41.092330    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c387219-4c629" event={"ID":"c1fa66d2-1746-4308-bc2a-1f909da2322e","Type":"ContainerStarted","Data":"f36618fea66e4b39d3fb311ab8968a98461451f3a0c1eb3cc73e703c6321b551"}
Nov 03 00:20:41.092958 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:41.092433    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c387219-4c629" event={"ID":"c1fa66d2-1746-4308-bc2a-1f909da2322e","Type":"ContainerStarted","Data":"2ff1b6051160552af377fd0aac5333466be7432fd9c949565bbb5d332a77e240"}
Nov 03 00:20:41.127296 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:41.127228    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c387219-4c629" podStartSLOduration=1.127211165 podStartE2EDuration="1.127211165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:41.126208203 +0000 UTC m=+924.005792246" watchObservedRunningTime="2026-11-03 00:20:41.127211165 +0000 UTC m=+924.006795211"
Nov 03 00:20:42.100802 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:42.100767    5012 generic.go:356] "Generic (PLEG): container finished" podID="c1fa66d2-1746-4308-bc2a-1f909da2322e" containerID="f36618fea66e4b39d3fb311ab8968a98461451f3a0c1eb3cc73e703c6321b551" exitCode=0
Nov 03 00:20:42.101227 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:42.100845    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c387219-4c629" event={"ID":"c1fa66d2-1746-4308-bc2a-1f909da2322e","Type":"ContainerDied","Data":"f36618fea66e4b39d3fb311ab8968a98461451f3a0c1eb3cc73e703c6321b551"}
Nov 03 00:20:43.262697 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:43.262664    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c387219-4c629"
Nov 03 00:20:43.419308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:43.419211    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1fa66d2-1746-4308-bc2a-1f909da2322e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1fa66d2-1746-4308-bc2a-1f909da2322e-maas-api-xks-ca\") pod \"c1fa66d2-1746-4308-bc2a-1f909da2322e\" (UID: \"c1fa66d2-1746-4308-bc2a-1f909da2322e\") "
Nov 03 00:20:43.419488 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:43.419360    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1fa66d2-1746-4308-bc2a-1f909da2322e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1fa66d2-1746-4308-bc2a-1f909da2322e-maas-api-openshift-ca\") pod \"c1fa66d2-1746-4308-bc2a-1f909da2322e\" (UID: \"c1fa66d2-1746-4308-bc2a-1f909da2322e\") "
Nov 03 00:20:43.419488 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:43.419417    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1fa66d2-1746-4308-bc2a-1f909da2322e-kube-api-access-mgs7v\" (UniqueName: \"kubernetes.io/projected/c1fa66d2-1746-4308-bc2a-1f909da2322e-kube-api-access-mgs7v\") pod \"c1fa66d2-1746-4308-bc2a-1f909da2322e\" (UID: \"c1fa66d2-1746-4308-bc2a-1f909da2322e\") "
Nov 03 00:20:43.419784 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:43.419755    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1fa66d2-1746-4308-bc2a-1f909da2322e-maas-api-openshift-ca" pod "c1fa66d2-1746-4308-bc2a-1f909da2322e" (UID: "c1fa66d2-1746-4308-bc2a-1f909da2322e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:20:43.421255 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:43.421235    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1fa66d2-1746-4308-bc2a-1f909da2322e-maas-api-xks-ca" pod "c1fa66d2-1746-4308-bc2a-1f909da2322e" (UID: "c1fa66d2-1746-4308-bc2a-1f909da2322e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:20:43.421375 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:43.421361    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1fa66d2-1746-4308-bc2a-1f909da2322e-kube-api-access-mgs7v" pod "c1fa66d2-1746-4308-bc2a-1f909da2322e" (UID: "c1fa66d2-1746-4308-bc2a-1f909da2322e"). InnerVolumeSpecName "kube-api-access-mgs7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:20:43.521197 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:43.521164    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgs7v\" (UniqueName: \"kubernetes.io/projected/c1fa66d2-1746-4308-bc2a-1f909da2322e-kube-api-access-mgs7v\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:43.521197 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:43.521195    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1fa66d2-1746-4308-bc2a-1f909da2322e-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:43.521197 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:43.521206    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1fa66d2-1746-4308-bc2a-1f909da2322e-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:44.117285 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.117254    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c387219-4c629"
Nov 03 00:20:44.117462 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.117292    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c387219-4c629" event={"ID":"c1fa66d2-1746-4308-bc2a-1f909da2322e","Type":"ContainerDied","Data":"2ff1b6051160552af377fd0aac5333466be7432fd9c949565bbb5d332a77e240"}
Nov 03 00:20:44.117462 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.117341    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ff1b6051160552af377fd0aac5333466be7432fd9c949565bbb5d332a77e240"
Nov 03 00:20:44.602593 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.602556    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"]
Nov 03 00:20:44.603959 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.603931    5012 cpu_manager.go:414] "containerMap: removing container" podUID="c1fa66d2-1746-4308-bc2a-1f909da2322e" containerName="revoke-keys"
Nov 03 00:20:44.603959 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.603957    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1fa66d2-1746-4308-bc2a-1f909da2322e" containerName="revoke-keys"
Nov 03 00:20:44.604161 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.604086    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1fa66d2-1746-4308-bc2a-1f909da2322e" containerName="revoke-keys"
Nov 03 00:20:44.608268 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.608245    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:20:44.623847 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.623820    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"]
Nov 03 00:20:44.625408 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.625387    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4c61c8\"/\"default-dockercfg-lqdnc\""
Nov 03 00:20:44.625547 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.625482    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4c61c8\"/\"distinct-w2-4c61c8-kserve-self-signed-certs\""
Nov 03 00:20:44.630102 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.630080    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-4c61c8\"/\"openshift-service-ca.crt\""
Nov 03 00:20:44.630102 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.630098    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-4c61c8\"/\"kube-root-ca.crt\""
Nov 03 00:20:44.736204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.736171    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-home\") pod \"distinct-w2-4c61c8-kserve-5c96989bd-zqhpm\" (UID: \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:20:44.736204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.736206    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-tls-certs\") pod \"distinct-w2-4c61c8-kserve-5c96989bd-zqhpm\" (UID: \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:20:44.736442 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.736250    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-tmp-dir\") pod \"distinct-w2-4c61c8-kserve-5c96989bd-zqhpm\" (UID: \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:20:44.736442 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.736375    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92p8l\" (UniqueName: \"kubernetes.io/projected/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-kube-api-access-92p8l\") pod \"distinct-w2-4c61c8-kserve-5c96989bd-zqhpm\" (UID: \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:20:44.736442 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.736425    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-dshm\") pod \"distinct-w2-4c61c8-kserve-5c96989bd-zqhpm\" (UID: \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:20:44.736608 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.736547    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-model-cache\") pod \"distinct-w2-4c61c8-kserve-5c96989bd-zqhpm\" (UID: \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:20:44.837884 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.837834    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-home\") pod \"distinct-w2-4c61c8-kserve-5c96989bd-zqhpm\" (UID: \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:20:44.838142 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.838122    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-tls-certs\") pod \"distinct-w2-4c61c8-kserve-5c96989bd-zqhpm\" (UID: \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:20:44.838274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.838262    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-tmp-dir\") pod \"distinct-w2-4c61c8-kserve-5c96989bd-zqhpm\" (UID: \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:20:44.838384 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.838371    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92p8l\" (UniqueName: \"kubernetes.io/projected/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-kube-api-access-92p8l\") pod \"distinct-w2-4c61c8-kserve-5c96989bd-zqhpm\" (UID: \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:20:44.838498 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.838487    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-dshm\") pod \"distinct-w2-4c61c8-kserve-5c96989bd-zqhpm\" (UID: \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:20:44.838640 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.838627    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-model-cache\") pod \"distinct-w2-4c61c8-kserve-5c96989bd-zqhpm\" (UID: \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:20:44.839042 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.838369    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-home\") pod \"distinct-w2-4c61c8-kserve-5c96989bd-zqhpm\" (UID: \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:20:44.839145 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.839070    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-tmp-dir\") pod \"distinct-w2-4c61c8-kserve-5c96989bd-zqhpm\" (UID: \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:20:44.839382 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.839359    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-model-cache\") pod \"distinct-w2-4c61c8-kserve-5c96989bd-zqhpm\" (UID: \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:20:44.842482 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.842454    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-dshm\") pod \"distinct-w2-4c61c8-kserve-5c96989bd-zqhpm\" (UID: \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:20:44.842837 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.842811    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-tls-certs\") pod \"distinct-w2-4c61c8-kserve-5c96989bd-zqhpm\" (UID: \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:20:44.856422 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.856319    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92p8l\" (UniqueName: \"kubernetes.io/projected/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-kube-api-access-92p8l\") pod \"distinct-w2-4c61c8-kserve-5c96989bd-zqhpm\" (UID: \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:20:44.924631 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:44.924596    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:20:45.225501 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:20:45.225476    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc91595ba_7c00_44a8_b6d5_2bb7499b54b3.slice/crio-8842a340ed04503630b72fca661985d18608ac918abe1389c99e56e68ce5a681 WatchSource:0}: Error finding container 8842a340ed04503630b72fca661985d18608ac918abe1389c99e56e68ce5a681: Status 404 returned error can't find the container with id 8842a340ed04503630b72fca661985d18608ac918abe1389c99e56e68ce5a681
Nov 03 00:20:45.225592 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:45.225498    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"]
Nov 03 00:20:46.134217 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:46.134179    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" event={"ID":"c91595ba-7c00-44a8-b6d5-2bb7499b54b3","Type":"ContainerStarted","Data":"eb88e654aa294725147144e20cb6b7381ce6888f17999650f12202ea91590c90"}
Nov 03 00:20:46.134629 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:46.134228    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" event={"ID":"c91595ba-7c00-44a8-b6d5-2bb7499b54b3","Type":"ContainerStarted","Data":"8842a340ed04503630b72fca661985d18608ac918abe1389c99e56e68ce5a681"}
Nov 03 00:20:46.171235 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:46.171186    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podStartSLOduration=2.171174531 podStartE2EDuration="2.171174531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:46.167028121 +0000 UTC m=+929.046612163" watchObservedRunningTime="2026-11-03 00:20:46.171174531 +0000 UTC m=+929.050758575"
Nov 03 00:20:46.925185 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:46.925151    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:20:48.271934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:48.268607    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"]
Nov 03 00:20:48.303933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:48.301792    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:48.332934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:48.312797    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"]
Nov 03 00:20:48.687934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:48.685530    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:48.687934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:48.685583    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8bb2\" (UniqueName: \"kubernetes.io/projected/76dcaa9e-a309-4329-8206-7d5ba46cc624-kube-api-access-f8bb2\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:48.687934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:48.685708    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:48.687934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:48.685755    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:48.687934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:48.685855    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:48.687934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:48.685886    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:48.687934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:48.685937    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:48.687934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:48.685985    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76dcaa9e-a309-4329-8206-7d5ba46cc624-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:48.687934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:48.686049    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:48.999002 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:48.937207    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:49.043935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.043037    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:49.043935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.043117    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:49.043935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.043181    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76dcaa9e-a309-4329-8206-7d5ba46cc624-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:49.043935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.043248    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:49.043935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.043366    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:49.043935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.043405    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8bb2\" (UniqueName: \"kubernetes.io/projected/76dcaa9e-a309-4329-8206-7d5ba46cc624-kube-api-access-f8bb2\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:49.043935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.043558    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:49.043935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.043623    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:49.044513 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.044268    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:49.044513 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:48.941532    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:49.044625 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.044601    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:49.057937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.044874    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:49.057937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.045893    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76dcaa9e-a309-4329-8206-7d5ba46cc624-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:49.214934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.190054    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:49.214934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.191561    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:49.214934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.192551    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:20:49.239940 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.229978    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"]
Nov 03 00:20:49.375933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.371227    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:49.375933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.371101    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8bb2\" (UniqueName: \"kubernetes.io/projected/76dcaa9e-a309-4329-8206-7d5ba46cc624-kube-api-access-f8bb2\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:49.379931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.378565    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.477769 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.477732    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4vzj\" (UniqueName: \"kubernetes.io/projected/626c13b1-be66-4342-ab51-20eb70cdd9a2-kube-api-access-w4vzj\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.478670 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.478122    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/626c13b1-be66-4342-ab51-20eb70cdd9a2-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.478670 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.478203    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.478670 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.478250    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.478670 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.478293    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.478670 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.478330    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.478670 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.478371    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.478670 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.478428    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.478670 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.478464    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.479248 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.477419    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:49.486128 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.486103    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"]
Nov 03 00:20:49.606108 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.605622    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"]
Nov 03 00:20:49.627310 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.627073    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.628262 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.628227    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.637000 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.627530    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.637752 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.637727    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4vzj\" (UniqueName: \"kubernetes.io/projected/626c13b1-be66-4342-ab51-20eb70cdd9a2-kube-api-access-w4vzj\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.638047 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.638027    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/626c13b1-be66-4342-ab51-20eb70cdd9a2-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.638280 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.638259    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.638438 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.638421    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.638596 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.638579    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.639075 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.639054    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.639240 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.639222    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.639758 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.639736    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.656002 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.654140    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/626c13b1-be66-4342-ab51-20eb70cdd9a2-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.656179 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.654391    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.656315 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.654645    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.678172 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.678143    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.706009 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.705971    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 03 00:20:49.743311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.743240    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.743311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.743298    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b34bfd-c225-46b7-adcc-2999244af328-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.743542 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.743338    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.743542 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.743370    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgdr8\" (UniqueName: \"kubernetes.io/projected/08b34bfd-c225-46b7-adcc-2999244af328-kube-api-access-zgdr8\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.743542 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.743476    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b34bfd-c225-46b7-adcc-2999244af328-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.743542 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.743519    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08b34bfd-c225-46b7-adcc-2999244af328-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.743746 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.743610    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.743746 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.743690    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.743846 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.743753    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.752373 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.749036    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"]
Nov 03 00:20:49.860936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.831642    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4vzj\" (UniqueName: \"kubernetes.io/projected/626c13b1-be66-4342-ab51-20eb70cdd9a2-kube-api-access-w4vzj\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.860936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.844712    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.860936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.844769    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgdr8\" (UniqueName: \"kubernetes.io/projected/08b34bfd-c225-46b7-adcc-2999244af328-kube-api-access-zgdr8\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.860936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.844877    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b34bfd-c225-46b7-adcc-2999244af328-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.860936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.844942    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08b34bfd-c225-46b7-adcc-2999244af328-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.860936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.845030    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.860936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.845105    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.860936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.845162    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.860936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.845247    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.860936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.845285    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b34bfd-c225-46b7-adcc-2999244af328-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.860936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.855376    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.860936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.856361    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b34bfd-c225-46b7-adcc-2999244af328-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.860936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.856885    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.860936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.857159    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.860936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.857166    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.860936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.857422    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.861901 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.861106    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.862807 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.862746    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b34bfd-c225-46b7-adcc-2999244af328-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.863691 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.863623    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:49.865683 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.865639    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:49.934318 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.932691    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:49.934318 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.933021    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 03 00:20:50.020048 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:49.998283    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:50.194994 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.193470    5012 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:50.352973 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.350800    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgdr8\" (UniqueName: \"kubernetes.io/projected/08b34bfd-c225-46b7-adcc-2999244af328-kube-api-access-zgdr8\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:50.387581 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.387483    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08b34bfd-c225-46b7-adcc-2999244af328-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:50.479462 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.479430    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:50.602828 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.602787    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"]
Nov 03 00:20:50.623415 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.623367    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e803ae6-h8t8f"]
Nov 03 00:20:50.643697 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:20:50.643621    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76dcaa9e_a309_4329_8206_7d5ba46cc624.slice/crio-343b5399db6874fb35a276a5db599a3b468151ed10242e7515b906e8fb25fca8 WatchSource:0}: Error finding container 343b5399db6874fb35a276a5db599a3b468151ed10242e7515b906e8fb25fca8: Status 404 returned error can't find the container with id 343b5399db6874fb35a276a5db599a3b468151ed10242e7515b906e8fb25fca8
Nov 03 00:20:50.649746 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.648240    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:50.649746 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.648330    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:50.649746 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.648375    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:50.649746 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.648948    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e803ae6-h8t8f"
Nov 03 00:20:50.650434 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.650404    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e803ae6-h8t8f"]
Nov 03 00:20:50.668501 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.668471    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 00:20:50.668843 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.668822    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:20:50.669107 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.669089    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 00:20:50.669258 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.669243    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:20:50.686820 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.685406    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88595eae-3638-43d7-83b9-77a1253b2879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e803ae6-h8t8f\" (UID: \"88595eae-3638-43d7-83b9-77a1253b2879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e803ae6-h8t8f"
Nov 03 00:20:50.686820 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.685567    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88595eae-3638-43d7-83b9-77a1253b2879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e803ae6-h8t8f\" (UID: \"88595eae-3638-43d7-83b9-77a1253b2879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e803ae6-h8t8f"
Nov 03 00:20:50.686820 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.685604    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6tf5\" (UniqueName: \"kubernetes.io/projected/88595eae-3638-43d7-83b9-77a1253b2879-kube-api-access-p6tf5\") pod \"maas-api-revoke-sub-8e803ae6-h8t8f\" (UID: \"88595eae-3638-43d7-83b9-77a1253b2879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e803ae6-h8t8f"
Nov 03 00:20:50.703770 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:20:50.703725    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod626c13b1_be66_4342_ab51_20eb70cdd9a2.slice/crio-ecfe4818676d949006b4c4283acaccc5f992115549fea5ff2e53b1efbcce5624 WatchSource:0}: Error finding container ecfe4818676d949006b4c4283acaccc5f992115549fea5ff2e53b1efbcce5624: Status 404 returned error can't find the container with id ecfe4818676d949006b4c4283acaccc5f992115549fea5ff2e53b1efbcce5624
Nov 03 00:20:50.711526 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.711494    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:50.711635 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.711559    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:50.711635 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.711591    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:50.715962 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.712780    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"]
Nov 03 00:20:50.786942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.786775    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88595eae-3638-43d7-83b9-77a1253b2879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e803ae6-h8t8f\" (UID: \"88595eae-3638-43d7-83b9-77a1253b2879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e803ae6-h8t8f"
Nov 03 00:20:50.786942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.786925    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88595eae-3638-43d7-83b9-77a1253b2879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e803ae6-h8t8f\" (UID: \"88595eae-3638-43d7-83b9-77a1253b2879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e803ae6-h8t8f"
Nov 03 00:20:50.787146 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.786971    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6tf5\" (UniqueName: \"kubernetes.io/projected/88595eae-3638-43d7-83b9-77a1253b2879-kube-api-access-p6tf5\") pod \"maas-api-revoke-sub-8e803ae6-h8t8f\" (UID: \"88595eae-3638-43d7-83b9-77a1253b2879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e803ae6-h8t8f"
Nov 03 00:20:50.787610 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.787589    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88595eae-3638-43d7-83b9-77a1253b2879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e803ae6-h8t8f\" (UID: \"88595eae-3638-43d7-83b9-77a1253b2879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e803ae6-h8t8f"
Nov 03 00:20:50.793498 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.793438    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88595eae-3638-43d7-83b9-77a1253b2879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e803ae6-h8t8f\" (UID: \"88595eae-3638-43d7-83b9-77a1253b2879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e803ae6-h8t8f"
Nov 03 00:20:50.805941 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.804389    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6tf5\" (UniqueName: \"kubernetes.io/projected/88595eae-3638-43d7-83b9-77a1253b2879-kube-api-access-p6tf5\") pod \"maas-api-revoke-sub-8e803ae6-h8t8f\" (UID: \"88595eae-3638-43d7-83b9-77a1253b2879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e803ae6-h8t8f"
Nov 03 00:20:50.943516 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.943483    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"]
Nov 03 00:20:50.944435 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:20:50.944398    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b34bfd_c225_46b7_adcc_2999244af328.slice/crio-f0f089238e5241219cf52065675efe5fe7d50cb4b28fecba5c1d1596f8e8325f WatchSource:0}: Error finding container f0f089238e5241219cf52065675efe5fe7d50cb4b28fecba5c1d1596f8e8325f: Status 404 returned error can't find the container with id f0f089238e5241219cf52065675efe5fe7d50cb4b28fecba5c1d1596f8e8325f
Nov 03 00:20:50.947319 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.947283    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:50.947418 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.947363    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:50.947418 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.947409    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:50.996496 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:50.996470    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e803ae6-h8t8f"
Nov 03 00:20:51.373974 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:51.373934    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e803ae6-h8t8f"]
Nov 03 00:20:51.473569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:51.473528    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e803ae6-h8t8f" event={"ID":"88595eae-3638-43d7-83b9-77a1253b2879","Type":"ContainerStarted","Data":"0986ff11fb6675ee150c2380a5b3eeaaa7f3b7a7362b0cd9c275749b8b516157"}
Nov 03 00:20:51.475317 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:51.475281    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x" event={"ID":"08b34bfd-c225-46b7-adcc-2999244af328","Type":"ContainerStarted","Data":"7115df497578842fb80ad63c681908cde635043019ae9a404fe591159d8981f4"}
Nov 03 00:20:51.475441 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:51.475344    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x" event={"ID":"08b34bfd-c225-46b7-adcc-2999244af328","Type":"ContainerStarted","Data":"f0f089238e5241219cf52065675efe5fe7d50cb4b28fecba5c1d1596f8e8325f"}
Nov 03 00:20:51.477470 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:51.477446    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb" event={"ID":"76dcaa9e-a309-4329-8206-7d5ba46cc624","Type":"ContainerStarted","Data":"01a8e50be35b4e9cc90343fcc77906c8d4f72cc638f3b008ec211a1b07e84739"}
Nov 03 00:20:51.477579 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:51.477500    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb" event={"ID":"76dcaa9e-a309-4329-8206-7d5ba46cc624","Type":"ContainerStarted","Data":"343b5399db6874fb35a276a5db599a3b468151ed10242e7515b906e8fb25fca8"}
Nov 03 00:20:51.479435 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:51.479414    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8" event={"ID":"626c13b1-be66-4342-ab51-20eb70cdd9a2","Type":"ContainerStarted","Data":"298b4d5fe81e474fbdd860e7fadf671b9979e3b1112c680e1f38f8e7c35dfe96"}
Nov 03 00:20:51.479519 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:51.479462    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8" event={"ID":"626c13b1-be66-4342-ab51-20eb70cdd9a2","Type":"ContainerStarted","Data":"ecfe4818676d949006b4c4283acaccc5f992115549fea5ff2e53b1efbcce5624"}
Nov 03 00:20:51.481462 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:51.481443    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:51.537678 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:51.537612    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x" podStartSLOduration=3.537592369 podStartE2EDuration="3.537592369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:51.531698162 +0000 UTC m=+934.411282206" watchObservedRunningTime="2026-11-03 00:20:51.537592369 +0000 UTC m=+934.417176414"
Nov 03 00:20:51.585657 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:51.585601    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8" podStartSLOduration=3.5855861129999997 podStartE2EDuration="3.585586113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:51.579830622 +0000 UTC m=+934.459414664" watchObservedRunningTime="2026-11-03 00:20:51.585586113 +0000 UTC m=+934.465170153"
Nov 03 00:20:51.663788 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:51.663655    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb" podStartSLOduration=3.6636322420000003 podStartE2EDuration="3.663632242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:51.656425307 +0000 UTC m=+934.536009351" watchObservedRunningTime="2026-11-03 00:20:51.663632242 +0000 UTC m=+934.543216287"
Nov 03 00:20:52.015523 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:52.015441    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:52.019374 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:52.019031    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 03 00:20:52.019374 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:52.019091    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8" podUID="626c13b1-be66-4342-ab51-20eb70cdd9a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 03 00:20:52.478053 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:52.478020    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:52.478923 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:52.478890    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 00:20:52.479011 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:52.478965    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb" podUID="76dcaa9e-a309-4329-8206-7d5ba46cc624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 00:20:52.481849 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:52.481827    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 00:20:52.481967 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:52.481866    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x" podUID="08b34bfd-c225-46b7-adcc-2999244af328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 00:20:52.528935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:52.528674    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e803ae6-h8t8f" event={"ID":"88595eae-3638-43d7-83b9-77a1253b2879","Type":"ContainerStarted","Data":"4859526eb15d1ca5565f1bc6696c98980e4bb62b960b37793da09c477e962128"}
Nov 03 00:20:52.571878 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:52.571825    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e803ae6-h8t8f" podStartSLOduration=2.57180753 podStartE2EDuration="2.57180753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:52.566037145 +0000 UTC m=+935.445621187" watchObservedRunningTime="2026-11-03 00:20:52.57180753 +0000 UTC m=+935.451391575"
Nov 03 00:20:53.005305 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:53.005272    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 03 00:20:53.005585 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:53.005556    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8" podUID="626c13b1-be66-4342-ab51-20eb70cdd9a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 03 00:20:53.478191 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:53.478157    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 00:20:53.478615 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:53.478211    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb" podUID="76dcaa9e-a309-4329-8206-7d5ba46cc624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 00:20:53.481453 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:53.481428    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 00:20:53.481571 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:53.481544    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x" podUID="08b34bfd-c225-46b7-adcc-2999244af328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 00:20:53.566285 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:53.566244    5012 generic.go:356] "Generic (PLEG): container finished" podID="88595eae-3638-43d7-83b9-77a1253b2879" containerID="4859526eb15d1ca5565f1bc6696c98980e4bb62b960b37793da09c477e962128" exitCode=0
Nov 03 00:20:53.566486 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:53.566328    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e803ae6-h8t8f" event={"ID":"88595eae-3638-43d7-83b9-77a1253b2879","Type":"ContainerDied","Data":"4859526eb15d1ca5565f1bc6696c98980e4bb62b960b37793da09c477e962128"}
Nov 03 00:20:54.018934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:54.016545    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 03 00:20:54.018934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:54.016606    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8" podUID="626c13b1-be66-4342-ab51-20eb70cdd9a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 03 00:20:54.478383 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:54.478339    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 00:20:54.478887 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:54.478396    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb" podUID="76dcaa9e-a309-4329-8206-7d5ba46cc624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 00:20:54.481155 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:54.481127    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 00:20:54.481275 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:54.481172    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x" podUID="08b34bfd-c225-46b7-adcc-2999244af328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 00:20:54.967934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:54.945975    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:20:55.019057 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:55.019020    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 03 00:20:55.019366 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:55.019318    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8" podUID="626c13b1-be66-4342-ab51-20eb70cdd9a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 03 00:20:55.095345 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:55.095315    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:20:55.176071 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:55.176031    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e803ae6-h8t8f"
Nov 03 00:20:55.371585 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:55.370408    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88595eae-3638-43d7-83b9-77a1253b2879-kube-api-access-p6tf5\" (UniqueName: \"kubernetes.io/projected/88595eae-3638-43d7-83b9-77a1253b2879-kube-api-access-p6tf5\") pod \"88595eae-3638-43d7-83b9-77a1253b2879\" (UID: \"88595eae-3638-43d7-83b9-77a1253b2879\") "
Nov 03 00:20:55.371585 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:55.370475    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88595eae-3638-43d7-83b9-77a1253b2879-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88595eae-3638-43d7-83b9-77a1253b2879-maas-api-xks-ca\") pod \"88595eae-3638-43d7-83b9-77a1253b2879\" (UID: \"88595eae-3638-43d7-83b9-77a1253b2879\") "
Nov 03 00:20:55.371585 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:55.370495    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88595eae-3638-43d7-83b9-77a1253b2879-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88595eae-3638-43d7-83b9-77a1253b2879-maas-api-openshift-ca\") pod \"88595eae-3638-43d7-83b9-77a1253b2879\" (UID: \"88595eae-3638-43d7-83b9-77a1253b2879\") "
Nov 03 00:20:55.371585 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:55.371263    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88595eae-3638-43d7-83b9-77a1253b2879-maas-api-openshift-ca" pod "88595eae-3638-43d7-83b9-77a1253b2879" (UID: "88595eae-3638-43d7-83b9-77a1253b2879"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:20:55.374927 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:55.374880    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88595eae-3638-43d7-83b9-77a1253b2879-kube-api-access-p6tf5" pod "88595eae-3638-43d7-83b9-77a1253b2879" (UID: "88595eae-3638-43d7-83b9-77a1253b2879"). InnerVolumeSpecName "kube-api-access-p6tf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:20:55.375115 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:55.375087    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88595eae-3638-43d7-83b9-77a1253b2879-maas-api-xks-ca" pod "88595eae-3638-43d7-83b9-77a1253b2879" (UID: "88595eae-3638-43d7-83b9-77a1253b2879"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:20:55.471683 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:55.471652    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6tf5\" (UniqueName: \"kubernetes.io/projected/88595eae-3638-43d7-83b9-77a1253b2879-kube-api-access-p6tf5\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:55.471683 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:55.471678    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88595eae-3638-43d7-83b9-77a1253b2879-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:55.471878 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:55.471687    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88595eae-3638-43d7-83b9-77a1253b2879-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:20:55.478230 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:55.478202    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 00:20:55.478324 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:55.478269    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb" podUID="76dcaa9e-a309-4329-8206-7d5ba46cc624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 00:20:55.480933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:55.480887    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 00:20:55.481241 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:55.480952    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x" podUID="08b34bfd-c225-46b7-adcc-2999244af328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 00:20:55.593149 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:55.593118    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e803ae6-h8t8f"
Nov 03 00:20:55.624007 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:55.595420    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e803ae6-h8t8f" event={"ID":"88595eae-3638-43d7-83b9-77a1253b2879","Type":"ContainerDied","Data":"0986ff11fb6675ee150c2380a5b3eeaaa7f3b7a7362b0cd9c275749b8b516157"}
Nov 03 00:20:55.624007 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:55.616014    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0986ff11fb6675ee150c2380a5b3eeaaa7f3b7a7362b0cd9c275749b8b516157"
Nov 03 00:20:55.677868 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:55.677844    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:20:56.006391 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:56.006299    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 03 00:20:56.006660 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:56.006632    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8" podUID="626c13b1-be66-4342-ab51-20eb70cdd9a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 03 00:20:56.478525 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:56.478488    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 00:20:56.478704 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:56.478569    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb" podUID="76dcaa9e-a309-4329-8206-7d5ba46cc624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 00:20:56.481873 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:56.481834    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 00:20:56.482268 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:56.481894    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x" podUID="08b34bfd-c225-46b7-adcc-2999244af328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 00:20:57.006307 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:57.006268    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 03 00:20:57.006605 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:57.006573    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8" podUID="626c13b1-be66-4342-ab51-20eb70cdd9a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 03 00:20:57.485301 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:57.485264    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 00:20:57.485881 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:57.485848    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb" podUID="76dcaa9e-a309-4329-8206-7d5ba46cc624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 00:20:57.486485 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:57.485279    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 00:20:57.486682 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:57.486657    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x" podUID="08b34bfd-c225-46b7-adcc-2999244af328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 00:20:58.333262 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:58.333233    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:58.334949 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:58.334902    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:58.338809 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:58.336429    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:20:58.539651 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:58.539583    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:58.540952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:58.540928    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:58.541774 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:58.541723    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:20:58.894106 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:58.894074    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:58.894590 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:58.894557    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:58.895307 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:58.895284    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:20:59.185295 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:59.185218    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:20:59.196820 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:20:59.196795    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:21:01.864776 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:01.864734    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:01.891163 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:01.891127    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:03.362930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.362885    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"]
Nov 03 00:21:03.363531 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.363516    5012 cpu_manager.go:414] "containerMap: removing container" podUID="88595eae-3638-43d7-83b9-77a1253b2879" containerName="revoke-keys"
Nov 03 00:21:03.363592 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.363533    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88595eae-3638-43d7-83b9-77a1253b2879" containerName="revoke-keys"
Nov 03 00:21:03.363642 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.363633    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="88595eae-3638-43d7-83b9-77a1253b2879" containerName="revoke-keys"
Nov 03 00:21:03.367211 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.367190    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.392970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.392946    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"]
Nov 03 00:21:03.400043 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.400017    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"]
Nov 03 00:21:03.410168 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.410141    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.421559 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.421533    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"]
Nov 03 00:21:03.457272 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.457236    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.457384 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.457277    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba191ff-7022-4265-a4fa-601ec997cb9b-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.457384 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.457325    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.457504 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.457387    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.457561 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.457535    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.457617 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.457598    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.457681 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.457667    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.457736 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.457719    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.457786 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.457744    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.457786 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.457769    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.457894 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.457834    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.457894 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.457862    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.457894 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.457887    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jmg2\" (UniqueName: \"kubernetes.io/projected/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-kube-api-access-6jmg2\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.458036 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.457961    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.458128 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.458103    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.458166 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.458156    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.458223 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.458208    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.458303 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.458290    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vnfw\" (UniqueName: \"kubernetes.io/projected/2ba191ff-7022-4265-a4fa-601ec997cb9b-kube-api-access-4vnfw\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.559697 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.559669    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.559821 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.559724    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.559821 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.559749    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.559821 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.559777    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.559821 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.559803    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.559821 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.559818    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.560061 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.559837    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.560061 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.559868    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.560264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.560230    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.560369 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.560265    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.560369 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.560353    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.560511 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.560382    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.560511 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.560387    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.560511 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.560426    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.560511 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.560443    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jmg2\" (UniqueName: \"kubernetes.io/projected/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-kube-api-access-6jmg2\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.560511 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.560472    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.560511 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.560494    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.560888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.560523    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.560888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.560552    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.560888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.560609    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vnfw\" (UniqueName: \"kubernetes.io/projected/2ba191ff-7022-4265-a4fa-601ec997cb9b-kube-api-access-4vnfw\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.560888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.560705    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.560888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.560736    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba191ff-7022-4265-a4fa-601ec997cb9b-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.560888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.560746    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.560888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.560763    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.561331 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.561134    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.561388 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.561357    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.561442 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.561358    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.561682 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.561662    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba191ff-7022-4265-a4fa-601ec997cb9b-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.562636 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.562612    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.562636 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.562633    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.562828 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.562808    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.563063 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.563045    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.577658 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.577629    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vnfw\" (UniqueName: \"kubernetes.io/projected/2ba191ff-7022-4265-a4fa-601ec997cb9b-kube-api-access-4vnfw\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.577742 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.577680    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.577784 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.577740    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.577952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.577902    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jmg2\" (UniqueName: \"kubernetes.io/projected/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-kube-api-access-6jmg2\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.681718 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.681666    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:03.727798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.727763    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:03.998082 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:03.998057    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"]
Nov 03 00:21:03.999642 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:21:03.999619    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba191ff_7022_4265_a4fa_601ec997cb9b.slice/crio-c1811678971e706937f8438cbf6828be9746760b5ac660ff8038ff73f79f5a04 WatchSource:0}: Error finding container c1811678971e706937f8438cbf6828be9746760b5ac660ff8038ff73f79f5a04: Status 404 returned error can't find the container with id c1811678971e706937f8438cbf6828be9746760b5ac660ff8038ff73f79f5a04
Nov 03 00:21:04.002080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:04.002039    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:04.002152 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:04.002114    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:04.002152 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:04.002144    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:04.055239 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:21:04.055214    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe8c9ae_0bb1_4073_b101_8c26d16b36d0.slice/crio-3e362d6aab30ab64a1556290b63ec6aa7552ee239f7da0008c0655d5a58c5887 WatchSource:0}: Error finding container 3e362d6aab30ab64a1556290b63ec6aa7552ee239f7da0008c0655d5a58c5887: Status 404 returned error can't find the container with id 3e362d6aab30ab64a1556290b63ec6aa7552ee239f7da0008c0655d5a58c5887
Nov 03 00:21:04.056276 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:04.056255    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"]
Nov 03 00:21:04.060689 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:04.059058    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:04.060689 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:04.059132    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:04.060689 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:04.059172    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:04.764567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:04.764520    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt" event={"ID":"2ba191ff-7022-4265-a4fa-601ec997cb9b","Type":"ContainerStarted","Data":"625e3236a9daacfa6f7fe124bb8f8e0f739786f5fbc2c6273758f409ee13e311"}
Nov 03 00:21:04.765640 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:04.765613    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt" event={"ID":"2ba191ff-7022-4265-a4fa-601ec997cb9b","Type":"ContainerStarted","Data":"c1811678971e706937f8438cbf6828be9746760b5ac660ff8038ff73f79f5a04"}
Nov 03 00:21:04.767579 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:04.767555    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf" event={"ID":"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0","Type":"ContainerStarted","Data":"d723e6020ac833528e82995db95681d48b4f8ebac9d9aa81440435c149c68fd9"}
Nov 03 00:21:04.767679 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:04.767593    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf" event={"ID":"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0","Type":"ContainerStarted","Data":"3e362d6aab30ab64a1556290b63ec6aa7552ee239f7da0008c0655d5a58c5887"}
Nov 03 00:21:04.822589 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:04.822525    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt" podStartSLOduration=1.822511967 podStartE2EDuration="1.822511967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:04.816784464 +0000 UTC m=+947.696368508" watchObservedRunningTime="2026-11-03 00:21:04.822511967 +0000 UTC m=+947.702096010"
Nov 03 00:21:04.858222 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:04.858169    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf" podStartSLOduration=1.858154576 podStartE2EDuration="1.858154576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:04.854532581 +0000 UTC m=+947.734116624" watchObservedRunningTime="2026-11-03 00:21:04.858154576 +0000 UTC m=+947.737738619"
Nov 03 00:21:05.644670 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:05.644624    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee42705a-jdn97"]
Nov 03 00:21:05.652435 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:05.651375    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee42705a-jdn97"
Nov 03 00:21:05.652435 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:05.652136    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee42705a-jdn97"]
Nov 03 00:21:05.673462 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:05.673436    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:21:05.673462 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:05.673454    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 00:21:05.673643 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:05.673501    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 00:21:05.673643 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:05.673531    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:21:05.682490 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:05.682468    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:05.684289 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:05.684265    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 00:21:05.684489 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:05.684462    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt" podUID="2ba191ff-7022-4265-a4fa-601ec997cb9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 00:21:05.735934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:05.730480    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:05.740440 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:05.740047    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 00:21:05.740440 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:05.740109    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf" podUID="fbe8c9ae-0bb1-4073-b101-8c26d16b36d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 00:21:05.846156 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:05.846123    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17d573e6-613e-404b-b4a0-9025239a7837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee42705a-jdn97\" (UID: \"17d573e6-613e-404b-b4a0-9025239a7837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee42705a-jdn97"
Nov 03 00:21:05.846644 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:05.846260    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7k2v\" (UniqueName: \"kubernetes.io/projected/17d573e6-613e-404b-b4a0-9025239a7837-kube-api-access-v7k2v\") pod \"maas-api-revoke-sub-ee42705a-jdn97\" (UID: \"17d573e6-613e-404b-b4a0-9025239a7837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee42705a-jdn97"
Nov 03 00:21:05.846644 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:05.846411    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17d573e6-613e-404b-b4a0-9025239a7837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee42705a-jdn97\" (UID: \"17d573e6-613e-404b-b4a0-9025239a7837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee42705a-jdn97"
Nov 03 00:21:05.947855 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:05.947771    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17d573e6-613e-404b-b4a0-9025239a7837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee42705a-jdn97\" (UID: \"17d573e6-613e-404b-b4a0-9025239a7837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee42705a-jdn97"
Nov 03 00:21:05.948060 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:05.947884    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17d573e6-613e-404b-b4a0-9025239a7837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee42705a-jdn97\" (UID: \"17d573e6-613e-404b-b4a0-9025239a7837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee42705a-jdn97"
Nov 03 00:21:05.948143 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:05.948112    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7k2v\" (UniqueName: \"kubernetes.io/projected/17d573e6-613e-404b-b4a0-9025239a7837-kube-api-access-v7k2v\") pod \"maas-api-revoke-sub-ee42705a-jdn97\" (UID: \"17d573e6-613e-404b-b4a0-9025239a7837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee42705a-jdn97"
Nov 03 00:21:05.948715 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:05.948687    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17d573e6-613e-404b-b4a0-9025239a7837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee42705a-jdn97\" (UID: \"17d573e6-613e-404b-b4a0-9025239a7837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee42705a-jdn97"
Nov 03 00:21:05.950418 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:05.950392    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17d573e6-613e-404b-b4a0-9025239a7837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee42705a-jdn97\" (UID: \"17d573e6-613e-404b-b4a0-9025239a7837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee42705a-jdn97"
Nov 03 00:21:05.964759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:05.964739    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7k2v\" (UniqueName: \"kubernetes.io/projected/17d573e6-613e-404b-b4a0-9025239a7837-kube-api-access-v7k2v\") pod \"maas-api-revoke-sub-ee42705a-jdn97\" (UID: \"17d573e6-613e-404b-b4a0-9025239a7837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee42705a-jdn97"
Nov 03 00:21:05.969320 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:05.969303    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee42705a-jdn97"
Nov 03 00:21:06.342629 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:06.342597    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee42705a-jdn97"]
Nov 03 00:21:06.682323 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:06.682176    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 00:21:06.682323 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:06.682236    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt" podUID="2ba191ff-7022-4265-a4fa-601ec997cb9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 00:21:06.732932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:06.732220    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 00:21:06.732932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:06.732282    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf" podUID="fbe8c9ae-0bb1-4073-b101-8c26d16b36d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 00:21:06.810930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:06.800794    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee42705a-jdn97" event={"ID":"17d573e6-613e-404b-b4a0-9025239a7837","Type":"ContainerStarted","Data":"386f843a0eda5dc4e88a0ffb7fca0b4d050f9e55142fb9a027e2b7b62eec4be9"}
Nov 03 00:21:06.810930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:06.800853    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee42705a-jdn97" event={"ID":"17d573e6-613e-404b-b4a0-9025239a7837","Type":"ContainerStarted","Data":"f0e860c687db9a1bd40395c5ec9efa56ea41bc075e2c6eabe9f34974c7e46d7e"}
Nov 03 00:21:06.846436 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:06.845994    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee42705a-jdn97" podStartSLOduration=1.845980316 podStartE2EDuration="1.845980316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:06.84421277 +0000 UTC m=+949.723796813" watchObservedRunningTime="2026-11-03 00:21:06.845980316 +0000 UTC m=+949.725564383"
Nov 03 00:21:07.364551 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:21:07.364461    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:21:07.682271 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:07.682185    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 00:21:07.682271 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:07.682232    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt" podUID="2ba191ff-7022-4265-a4fa-601ec997cb9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 00:21:07.728931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:07.728236    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 00:21:07.728931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:07.728292    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf" podUID="fbe8c9ae-0bb1-4073-b101-8c26d16b36d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 00:21:07.855441 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:07.855392    5012 generic.go:356] "Generic (PLEG): container finished" podID="17d573e6-613e-404b-b4a0-9025239a7837" containerID="386f843a0eda5dc4e88a0ffb7fca0b4d050f9e55142fb9a027e2b7b62eec4be9" exitCode=0
Nov 03 00:21:07.855945 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:07.855620    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee42705a-jdn97" event={"ID":"17d573e6-613e-404b-b4a0-9025239a7837","Type":"ContainerDied","Data":"386f843a0eda5dc4e88a0ffb7fca0b4d050f9e55142fb9a027e2b7b62eec4be9"}
Nov 03 00:21:08.682940 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:08.682883    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 00:21:08.683165 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:08.683141    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt" podUID="2ba191ff-7022-4265-a4fa-601ec997cb9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 00:21:08.735365 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:08.735324    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 00:21:08.735633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:08.735605    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf" podUID="fbe8c9ae-0bb1-4073-b101-8c26d16b36d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 00:21:09.172112 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:09.172084    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee42705a-jdn97"
Nov 03 00:21:09.220236 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:09.201330    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17d573e6-613e-404b-b4a0-9025239a7837-kube-api-access-v7k2v\" (UniqueName: \"kubernetes.io/projected/17d573e6-613e-404b-b4a0-9025239a7837-kube-api-access-v7k2v\") pod \"17d573e6-613e-404b-b4a0-9025239a7837\" (UID: \"17d573e6-613e-404b-b4a0-9025239a7837\") "
Nov 03 00:21:09.220236 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:09.201489    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17d573e6-613e-404b-b4a0-9025239a7837-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17d573e6-613e-404b-b4a0-9025239a7837-maas-api-openshift-ca\") pod \"17d573e6-613e-404b-b4a0-9025239a7837\" (UID: \"17d573e6-613e-404b-b4a0-9025239a7837\") "
Nov 03 00:21:09.220236 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:09.201652    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17d573e6-613e-404b-b4a0-9025239a7837-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17d573e6-613e-404b-b4a0-9025239a7837-maas-api-xks-ca\") pod \"17d573e6-613e-404b-b4a0-9025239a7837\" (UID: \"17d573e6-613e-404b-b4a0-9025239a7837\") "
Nov 03 00:21:09.222030 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:09.220901    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d573e6-613e-404b-b4a0-9025239a7837-maas-api-openshift-ca" pod "17d573e6-613e-404b-b4a0-9025239a7837" (UID: "17d573e6-613e-404b-b4a0-9025239a7837"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:21:09.251251 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:09.251138    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d573e6-613e-404b-b4a0-9025239a7837-kube-api-access-v7k2v" pod "17d573e6-613e-404b-b4a0-9025239a7837" (UID: "17d573e6-613e-404b-b4a0-9025239a7837"). InnerVolumeSpecName "kube-api-access-v7k2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:21:09.251251 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:09.251216    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d573e6-613e-404b-b4a0-9025239a7837-maas-api-xks-ca" pod "17d573e6-613e-404b-b4a0-9025239a7837" (UID: "17d573e6-613e-404b-b4a0-9025239a7837"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:21:09.312946 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:09.311943    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17d573e6-613e-404b-b4a0-9025239a7837-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:21:09.312946 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:09.311981    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7k2v\" (UniqueName: \"kubernetes.io/projected/17d573e6-613e-404b-b4a0-9025239a7837-kube-api-access-v7k2v\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:21:09.312946 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:09.311999    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17d573e6-613e-404b-b4a0-9025239a7837-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:21:09.682335 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:09.682291    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 00:21:09.682503 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:09.682351    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt" podUID="2ba191ff-7022-4265-a4fa-601ec997cb9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 00:21:09.733933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:09.731263    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 00:21:09.733933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:09.731322    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf" podUID="fbe8c9ae-0bb1-4073-b101-8c26d16b36d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 00:21:09.882406 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:09.882370    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee42705a-jdn97" event={"ID":"17d573e6-613e-404b-b4a0-9025239a7837","Type":"ContainerDied","Data":"f0e860c687db9a1bd40395c5ec9efa56ea41bc075e2c6eabe9f34974c7e46d7e"}
Nov 03 00:21:09.882578 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:09.882431    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0e860c687db9a1bd40395c5ec9efa56ea41bc075e2c6eabe9f34974c7e46d7e"
Nov 03 00:21:09.882578 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:09.882494    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee42705a-jdn97"
Nov 03 00:21:10.682834 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:10.682792    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 00:21:10.683210 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:10.682865    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt" podUID="2ba191ff-7022-4265-a4fa-601ec997cb9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 00:21:10.728716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:10.728678    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 00:21:10.728878 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:10.728736    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf" podUID="fbe8c9ae-0bb1-4073-b101-8c26d16b36d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 00:21:11.733041 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:11.733008    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:11.733554 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:11.733403    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:11.734304 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:11.734285    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:21:11.949542 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:11.949507    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:11.950049 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:11.950024    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:11.951369 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:11.951347    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:21:15.788923 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:15.788884    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c2ce0e7-p4vkh"]
Nov 03 00:21:15.789530 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:15.789516    5012 cpu_manager.go:414] "containerMap: removing container" podUID="17d573e6-613e-404b-b4a0-9025239a7837" containerName="revoke-keys"
Nov 03 00:21:15.789583 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:15.789531    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17d573e6-613e-404b-b4a0-9025239a7837" containerName="revoke-keys"
Nov 03 00:21:15.789652 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:15.789643    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="17d573e6-613e-404b-b4a0-9025239a7837" containerName="revoke-keys"
Nov 03 00:21:15.794665 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:15.794641    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2ce0e7-p4vkh"
Nov 03 00:21:15.811880 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:15.811855    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c2ce0e7-p4vkh"]
Nov 03 00:21:15.814102 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:15.814079    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:21:15.814807 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:15.814781    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:21:15.814917 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:15.814822    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 00:21:15.828751 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:15.828723    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 00:21:15.959396 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:15.921384    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s462f\" (UniqueName: \"kubernetes.io/projected/2a518260-33fc-47ab-b22a-9b716fe73ae3-kube-api-access-s462f\") pod \"maas-api-revoke-sub-9c2ce0e7-p4vkh\" (UID: \"2a518260-33fc-47ab-b22a-9b716fe73ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2ce0e7-p4vkh"
Nov 03 00:21:15.960525 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:15.960489    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a518260-33fc-47ab-b22a-9b716fe73ae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c2ce0e7-p4vkh\" (UID: \"2a518260-33fc-47ab-b22a-9b716fe73ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2ce0e7-p4vkh"
Nov 03 00:21:15.960945 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:15.960892    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a518260-33fc-47ab-b22a-9b716fe73ae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c2ce0e7-p4vkh\" (UID: \"2a518260-33fc-47ab-b22a-9b716fe73ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2ce0e7-p4vkh"
Nov 03 00:21:16.062037 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:16.061952    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a518260-33fc-47ab-b22a-9b716fe73ae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c2ce0e7-p4vkh\" (UID: \"2a518260-33fc-47ab-b22a-9b716fe73ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2ce0e7-p4vkh"
Nov 03 00:21:16.062221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:16.062035    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a518260-33fc-47ab-b22a-9b716fe73ae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c2ce0e7-p4vkh\" (UID: \"2a518260-33fc-47ab-b22a-9b716fe73ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2ce0e7-p4vkh"
Nov 03 00:21:16.062221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:16.062143    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s462f\" (UniqueName: \"kubernetes.io/projected/2a518260-33fc-47ab-b22a-9b716fe73ae3-kube-api-access-s462f\") pod \"maas-api-revoke-sub-9c2ce0e7-p4vkh\" (UID: \"2a518260-33fc-47ab-b22a-9b716fe73ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2ce0e7-p4vkh"
Nov 03 00:21:16.062738 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:16.062707    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a518260-33fc-47ab-b22a-9b716fe73ae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c2ce0e7-p4vkh\" (UID: \"2a518260-33fc-47ab-b22a-9b716fe73ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2ce0e7-p4vkh"
Nov 03 00:21:16.065091 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:16.065051    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a518260-33fc-47ab-b22a-9b716fe73ae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c2ce0e7-p4vkh\" (UID: \"2a518260-33fc-47ab-b22a-9b716fe73ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2ce0e7-p4vkh"
Nov 03 00:21:16.079468 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:16.079440    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s462f\" (UniqueName: \"kubernetes.io/projected/2a518260-33fc-47ab-b22a-9b716fe73ae3-kube-api-access-s462f\") pod \"maas-api-revoke-sub-9c2ce0e7-p4vkh\" (UID: \"2a518260-33fc-47ab-b22a-9b716fe73ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2ce0e7-p4vkh"
Nov 03 00:21:16.114113 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:16.114086    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2ce0e7-p4vkh"
Nov 03 00:21:16.697993 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:16.697855    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c2ce0e7-p4vkh"]
Nov 03 00:21:16.766970 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:21:16.766900    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a518260_33fc_47ab_b22a_9b716fe73ae3.slice/crio-c670ae7d07bba0a083f552b8b83243ccafb9b4e6835e0b5bf57766df69146166 WatchSource:0}: Error finding container c670ae7d07bba0a083f552b8b83243ccafb9b4e6835e0b5bf57766df69146166: Status 404 returned error can't find the container with id c670ae7d07bba0a083f552b8b83243ccafb9b4e6835e0b5bf57766df69146166
Nov 03 00:21:17.055091 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:17.055054    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2ce0e7-p4vkh" event={"ID":"2a518260-33fc-47ab-b22a-9b716fe73ae3","Type":"ContainerStarted","Data":"c670ae7d07bba0a083f552b8b83243ccafb9b4e6835e0b5bf57766df69146166"}
Nov 03 00:21:17.635007 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:17.613502    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:17.636612 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:17.615171    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:17.916989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:17.916431    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:18.205261 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:18.205148    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:18.466024 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:21:18.451998    5012 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a518260_33fc_47ab_b22a_9b716fe73ae3.slice/crio-conmon-8df4439482b63bc7be24fdec7798ebb9cb86526076495b600b014c86a340d5c5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a518260_33fc_47ab_b22a_9b716fe73ae3.slice/crio-8df4439482b63bc7be24fdec7798ebb9cb86526076495b600b014c86a340d5c5.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 00:21:18.485531 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:21:18.456057    5012 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a518260_33fc_47ab_b22a_9b716fe73ae3.slice/crio-8df4439482b63bc7be24fdec7798ebb9cb86526076495b600b014c86a340d5c5.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 00:21:18.640933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:18.640111    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:18.640933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:18.640587    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:18.641346 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:18.641243    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:21:18.641346 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:18.641305    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:21:18.840988 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:18.840047    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:21:18.916269 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:18.915995    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 03 00:21:19.046939 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:19.043507    5012 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": context deadline exceeded"
Nov 03 00:21:19.059060 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:19.059023    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2ce0e7-p4vkh" event={"ID":"2a518260-33fc-47ab-b22a-9b716fe73ae3","Type":"ContainerStarted","Data":"8df4439482b63bc7be24fdec7798ebb9cb86526076495b600b014c86a340d5c5"}
Nov 03 00:21:19.302935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:19.302331    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 03 00:21:19.420053 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:19.420002    5012 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:19.533878 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:19.492777    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:21:19.541955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:19.541886    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:21:19.681930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:19.666342    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:19.903533 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:19.903498    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:21:20.081885 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:20.081753    5012 generic.go:356] "Generic (PLEG): container finished" podID="2a518260-33fc-47ab-b22a-9b716fe73ae3" containerID="8df4439482b63bc7be24fdec7798ebb9cb86526076495b600b014c86a340d5c5" exitCode=0
Nov 03 00:21:20.082093 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:20.081878    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2ce0e7-p4vkh" event={"ID":"2a518260-33fc-47ab-b22a-9b716fe73ae3","Type":"ContainerDied","Data":"8df4439482b63bc7be24fdec7798ebb9cb86526076495b600b014c86a340d5c5"}
Nov 03 00:21:20.266090 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:20.266064    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2ce0e7-p4vkh"
Nov 03 00:21:20.312152 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:20.312121    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a518260-33fc-47ab-b22a-9b716fe73ae3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a518260-33fc-47ab-b22a-9b716fe73ae3-maas-api-xks-ca\") pod \"2a518260-33fc-47ab-b22a-9b716fe73ae3\" (UID: \"2a518260-33fc-47ab-b22a-9b716fe73ae3\") "
Nov 03 00:21:20.312525 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:20.312179    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a518260-33fc-47ab-b22a-9b716fe73ae3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a518260-33fc-47ab-b22a-9b716fe73ae3-maas-api-openshift-ca\") pod \"2a518260-33fc-47ab-b22a-9b716fe73ae3\" (UID: \"2a518260-33fc-47ab-b22a-9b716fe73ae3\") "
Nov 03 00:21:20.312525 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:20.312391    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a518260-33fc-47ab-b22a-9b716fe73ae3-kube-api-access-s462f\" (UniqueName: \"kubernetes.io/projected/2a518260-33fc-47ab-b22a-9b716fe73ae3-kube-api-access-s462f\") pod \"2a518260-33fc-47ab-b22a-9b716fe73ae3\" (UID: \"2a518260-33fc-47ab-b22a-9b716fe73ae3\") "
Nov 03 00:21:20.312656 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:20.312616    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a518260-33fc-47ab-b22a-9b716fe73ae3-maas-api-openshift-ca" pod "2a518260-33fc-47ab-b22a-9b716fe73ae3" (UID: "2a518260-33fc-47ab-b22a-9b716fe73ae3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:21:20.312816 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:20.312792    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a518260-33fc-47ab-b22a-9b716fe73ae3-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:21:20.314281 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:20.314260    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a518260-33fc-47ab-b22a-9b716fe73ae3-maas-api-xks-ca" pod "2a518260-33fc-47ab-b22a-9b716fe73ae3" (UID: "2a518260-33fc-47ab-b22a-9b716fe73ae3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:21:20.314458 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:20.314441    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a518260-33fc-47ab-b22a-9b716fe73ae3-kube-api-access-s462f" pod "2a518260-33fc-47ab-b22a-9b716fe73ae3" (UID: "2a518260-33fc-47ab-b22a-9b716fe73ae3"). InnerVolumeSpecName "kube-api-access-s462f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:21:20.414113 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:20.414087    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a518260-33fc-47ab-b22a-9b716fe73ae3-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:21:20.414113 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:20.414112    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s462f\" (UniqueName: \"kubernetes.io/projected/2a518260-33fc-47ab-b22a-9b716fe73ae3-kube-api-access-s462f\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:21:21.090877 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:21.090846    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2ce0e7-p4vkh"
Nov 03 00:21:21.090877 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:21.090859    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2ce0e7-p4vkh" event={"ID":"2a518260-33fc-47ab-b22a-9b716fe73ae3","Type":"ContainerDied","Data":"c670ae7d07bba0a083f552b8b83243ccafb9b4e6835e0b5bf57766df69146166"}
Nov 03 00:21:21.091098 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:21.090905    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c670ae7d07bba0a083f552b8b83243ccafb9b4e6835e0b5bf57766df69146166"
Nov 03 00:21:24.766649 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:24.766617    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b769d3f7-dt4q4"]
Nov 03 00:21:24.767287 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:24.767272    5012 cpu_manager.go:414] "containerMap: removing container" podUID="2a518260-33fc-47ab-b22a-9b716fe73ae3" containerName="revoke-keys"
Nov 03 00:21:24.767328 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:24.767288    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a518260-33fc-47ab-b22a-9b716fe73ae3" containerName="revoke-keys"
Nov 03 00:21:24.767391 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:24.767382    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a518260-33fc-47ab-b22a-9b716fe73ae3" containerName="revoke-keys"
Nov 03 00:21:24.772266 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:24.772248    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b769d3f7-dt4q4"
Nov 03 00:21:24.786789 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:24.786763    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 00:21:24.786789 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:24.786774    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 00:21:24.790154 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:24.790131    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b769d3f7-dt4q4"]
Nov 03 00:21:24.794233 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:24.794210    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:21:24.794374 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:24.794357    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:21:24.868349 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:24.868316    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4jjf\" (UniqueName: \"kubernetes.io/projected/86d3e235-326b-46d9-b062-f37a3d94b661-kube-api-access-f4jjf\") pod \"maas-api-revoke-sub-b769d3f7-dt4q4\" (UID: \"86d3e235-326b-46d9-b062-f37a3d94b661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b769d3f7-dt4q4"
Nov 03 00:21:24.868532 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:24.868514    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d3e235-326b-46d9-b062-f37a3d94b661-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b769d3f7-dt4q4\" (UID: \"86d3e235-326b-46d9-b062-f37a3d94b661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b769d3f7-dt4q4"
Nov 03 00:21:24.868832 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:24.868570    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d3e235-326b-46d9-b062-f37a3d94b661-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b769d3f7-dt4q4\" (UID: \"86d3e235-326b-46d9-b062-f37a3d94b661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b769d3f7-dt4q4"
Nov 03 00:21:24.970229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:24.970195    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d3e235-326b-46d9-b062-f37a3d94b661-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b769d3f7-dt4q4\" (UID: \"86d3e235-326b-46d9-b062-f37a3d94b661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b769d3f7-dt4q4"
Nov 03 00:21:24.970419 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:24.970261    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d3e235-326b-46d9-b062-f37a3d94b661-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b769d3f7-dt4q4\" (UID: \"86d3e235-326b-46d9-b062-f37a3d94b661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b769d3f7-dt4q4"
Nov 03 00:21:24.970419 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:24.970345    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4jjf\" (UniqueName: \"kubernetes.io/projected/86d3e235-326b-46d9-b062-f37a3d94b661-kube-api-access-f4jjf\") pod \"maas-api-revoke-sub-b769d3f7-dt4q4\" (UID: \"86d3e235-326b-46d9-b062-f37a3d94b661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b769d3f7-dt4q4"
Nov 03 00:21:24.971653 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:24.971610    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d3e235-326b-46d9-b062-f37a3d94b661-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b769d3f7-dt4q4\" (UID: \"86d3e235-326b-46d9-b062-f37a3d94b661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b769d3f7-dt4q4"
Nov 03 00:21:24.975388 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:24.975361    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d3e235-326b-46d9-b062-f37a3d94b661-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b769d3f7-dt4q4\" (UID: \"86d3e235-326b-46d9-b062-f37a3d94b661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b769d3f7-dt4q4"
Nov 03 00:21:24.988058 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:24.988021    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4jjf\" (UniqueName: \"kubernetes.io/projected/86d3e235-326b-46d9-b062-f37a3d94b661-kube-api-access-f4jjf\") pod \"maas-api-revoke-sub-b769d3f7-dt4q4\" (UID: \"86d3e235-326b-46d9-b062-f37a3d94b661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b769d3f7-dt4q4"
Nov 03 00:21:25.086951 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:25.086826    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b769d3f7-dt4q4"
Nov 03 00:21:25.389058 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:21:25.389021    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d3e235_326b_46d9_b062_f37a3d94b661.slice/crio-594f060b5500d42313fc373a390f13b60c665f96223734c138872750304b219d WatchSource:0}: Error finding container 594f060b5500d42313fc373a390f13b60c665f96223734c138872750304b219d: Status 404 returned error can't find the container with id 594f060b5500d42313fc373a390f13b60c665f96223734c138872750304b219d
Nov 03 00:21:25.389828 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:25.389800    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b769d3f7-dt4q4"]
Nov 03 00:21:26.134398 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:26.134355    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b769d3f7-dt4q4" event={"ID":"86d3e235-326b-46d9-b062-f37a3d94b661","Type":"ContainerStarted","Data":"6e66032a415d09bfd1f61c2fce255e8af3fea1d92a68024b2d419880acaeed97"}
Nov 03 00:21:26.134796 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:26.134428    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b769d3f7-dt4q4" event={"ID":"86d3e235-326b-46d9-b062-f37a3d94b661","Type":"ContainerStarted","Data":"594f060b5500d42313fc373a390f13b60c665f96223734c138872750304b219d"}
Nov 03 00:21:26.170524 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:26.170471    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b769d3f7-dt4q4" podStartSLOduration=2.170456412 podStartE2EDuration="2.170456412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:26.166735249 +0000 UTC m=+969.046319294" watchObservedRunningTime="2026-11-03 00:21:26.170456412 +0000 UTC m=+969.050040455"
Nov 03 00:21:27.172845 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:27.172803    5012 generic.go:356] "Generic (PLEG): container finished" podID="86d3e235-326b-46d9-b062-f37a3d94b661" containerID="6e66032a415d09bfd1f61c2fce255e8af3fea1d92a68024b2d419880acaeed97" exitCode=0
Nov 03 00:21:27.182060 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:27.182023    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b769d3f7-dt4q4" event={"ID":"86d3e235-326b-46d9-b062-f37a3d94b661","Type":"ContainerDied","Data":"6e66032a415d09bfd1f61c2fce255e8af3fea1d92a68024b2d419880acaeed97"}
Nov 03 00:21:28.818933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:28.814018    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:28.844073 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:28.830943    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:28.872934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:28.861147    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Nov 03 00:21:28.928255 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:28.915833    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:29.593934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:29.588639    5012 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": context deadline exceeded"
Nov 03 00:21:29.619470 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:29.619423    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 03 00:21:29.620522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:29.620490    5012 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded"
Nov 03 00:21:29.875933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:29.872190    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:29.875933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:29.875177    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 03 00:21:29.875933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:29.875525    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"]
Nov 03 00:21:29.963934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:29.929846    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:29.963934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:29.954655    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 03 00:21:30.394259 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.394226    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b769d3f7-dt4q4"
Nov 03 00:21:30.403936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.399669    5012 cpu_manager.go:414] "containerMap: removing container" podUID="86d3e235-326b-46d9-b062-f37a3d94b661" containerName="revoke-keys"
Nov 03 00:21:30.403936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.399703    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86d3e235-326b-46d9-b062-f37a3d94b661" containerName="revoke-keys"
Nov 03 00:21:30.403936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.400138    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="86d3e235-326b-46d9-b062-f37a3d94b661" containerName="revoke-keys"
Nov 03 00:21:30.421978 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.419555    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d3e235-326b-46d9-b062-f37a3d94b661-kube-api-access-f4jjf\" (UniqueName: \"kubernetes.io/projected/86d3e235-326b-46d9-b062-f37a3d94b661-kube-api-access-f4jjf\") pod \"86d3e235-326b-46d9-b062-f37a3d94b661\" (UID: \"86d3e235-326b-46d9-b062-f37a3d94b661\") "
Nov 03 00:21:30.421978 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.419742    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86d3e235-326b-46d9-b062-f37a3d94b661-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d3e235-326b-46d9-b062-f37a3d94b661-maas-api-xks-ca\") pod \"86d3e235-326b-46d9-b062-f37a3d94b661\" (UID: \"86d3e235-326b-46d9-b062-f37a3d94b661\") "
Nov 03 00:21:30.421978 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.419797    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86d3e235-326b-46d9-b062-f37a3d94b661-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d3e235-326b-46d9-b062-f37a3d94b661-maas-api-openshift-ca\") pod \"86d3e235-326b-46d9-b062-f37a3d94b661\" (UID: \"86d3e235-326b-46d9-b062-f37a3d94b661\") "
Nov 03 00:21:30.428241 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.428209    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"]
Nov 03 00:21:30.435939 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.428525    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:21:30.436264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.436240    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:21:30.436500 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.436484    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:21:30.442071 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.442045    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:21:30.451933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.442743    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:30.503791 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.503510    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9f76b708-openshift-default-dockercfg-84tf6\""
Nov 03 00:21:30.579938 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.576064    5012 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:30.579938 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.579786    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-workload-socket\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:30.579938 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.579843    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-envoy\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:30.604301 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.604265    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-data\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:30.604564 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.604541    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-podinfo\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:30.604796 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.604777    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-token\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:30.604943 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.604921    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istiod-ca-cert\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:30.605183 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.605158    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlbvb\" (UniqueName: \"kubernetes.io/projected/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-kube-api-access-qlbvb\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:30.606593 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.606569    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-credential-socket\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:30.618933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.606785    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-workload-certs\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:30.651400 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.644163    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 03 00:21:30.775935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.750569    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-workload-certs\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:30.803237 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.803202    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-workload-socket\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:30.803437 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.803289    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-envoy\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:30.803437 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.803392    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-data\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:30.803566 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.803473    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-podinfo\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:30.803624 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.803593    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-token\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:30.803682 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.803634    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istiod-ca-cert\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:30.828933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.803753    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlbvb\" (UniqueName: \"kubernetes.io/projected/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-kube-api-access-qlbvb\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:30.828933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.803971    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-credential-socket\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:30.830347 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.767552    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-workload-certs\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:30.832404 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.832375    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-workload-socket\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:30.858933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.831995    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-credential-socket\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:30.877374 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.877344    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-data\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:30.877927 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.776365    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d3e235-326b-46d9-b062-f37a3d94b661-maas-api-openshift-ca" pod "86d3e235-326b-46d9-b062-f37a3d94b661" (UID: "86d3e235-326b-46d9-b062-f37a3d94b661"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:21:30.919512 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.919435    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istiod-ca-cert\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:30.931936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.770388    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:21:30.939958 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.930062    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:30.975042 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.974990    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:31.006959 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.999328    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:31.018933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.999388    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Nov 03 00:21:31.170682 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:31.170609    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-envoy\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:31.289674 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:31.047050    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86d3e235-326b-46d9-b062-f37a3d94b661-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:21:31.289869 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:31.177144    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlbvb\" (UniqueName: \"kubernetes.io/projected/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-kube-api-access-qlbvb\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:31.289999 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:30.999464    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:21:31.290184 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:31.290156    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl" podUID="34b0a0f0-c396-4b0d-8f77-f032a1174374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:31.339935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:31.289611    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-podinfo\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:31.339935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:31.289595    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-token\") pod \"e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") " pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:31.339935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:31.176449    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86d3e235-326b-46d9-b062-f37a3d94b661-maas-api-xks-ca" pod "86d3e235-326b-46d9-b062-f37a3d94b661" (UID: "86d3e235-326b-46d9-b062-f37a3d94b661"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:21:31.339935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:31.214244    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d3e235-326b-46d9-b062-f37a3d94b661-kube-api-access-f4jjf" pod "86d3e235-326b-46d9-b062-f37a3d94b661" (UID: "86d3e235-326b-46d9-b062-f37a3d94b661"). InnerVolumeSpecName "kube-api-access-f4jjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:21:31.457989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:31.434486    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b769d3f7-dt4q4"
Nov 03 00:21:31.457989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:31.447487    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4jjf\" (UniqueName: \"kubernetes.io/projected/86d3e235-326b-46d9-b062-f37a3d94b661-kube-api-access-f4jjf\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:21:31.457989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:31.448272    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86d3e235-326b-46d9-b062-f37a3d94b661-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:21:31.603935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:31.544995    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:31.847085 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:31.748338    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:31.853461 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:31.853425    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b769d3f7-dt4q4" event={"ID":"86d3e235-326b-46d9-b062-f37a3d94b661","Type":"ContainerDied","Data":"594f060b5500d42313fc373a390f13b60c665f96223734c138872750304b219d"}
Nov 03 00:21:31.853670 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:31.853649    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="594f060b5500d42313fc373a390f13b60c665f96223734c138872750304b219d"
Nov 03 00:21:31.954019 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:31.953964    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:32.093396 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:32.021007    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:32.093396 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:32.058594    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Nov 03 00:21:32.186995 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:32.159314    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:32.818934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:32.798719    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 03 00:21:32.981933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:32.981026    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:33.035933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:33.029589    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:33.061242 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:33.061199    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:33.413933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:33.413885    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:21:33.440981 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:33.440080    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:21:33.459944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:33.442078    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 03 00:21:33.743713 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:33.743684    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:21:33.743933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:33.743868    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:33.744045 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:33.744031    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:21:33.744199 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:33.744186    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:21:33.745375 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:33.744045    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:33.745489 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:33.745453    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:33.778002 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:33.777972    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"]
Nov 03 00:21:33.962753 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:33.962715    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5" event={"ID":"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa","Type":"ContainerStarted","Data":"6bab4d0ce3444e7ed0526108ea75429cf427b9fff7c744793d938bf1bd2ac85c"}
Nov 03 00:21:33.962962 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:33.962775    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5" event={"ID":"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa","Type":"ContainerStarted","Data":"ca7ff1fc428246267addc730e6292da81868d157de3b1757d7352554d1343876"}
Nov 03 00:21:34.018697 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:34.018588    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5" podStartSLOduration=5.018568939 podStartE2EDuration="5.018568939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:34.012809462 +0000 UTC m=+976.892393506" watchObservedRunningTime="2026-11-03 00:21:34.018568939 +0000 UTC m=+976.898152983"
Nov 03 00:21:34.578105 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:34.578070    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:34.583593 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:34.583564    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:34.971682 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:34.971650    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:34.972794 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:34.972774    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:21:37.364093 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:21:37.364040    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:21:38.476467 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:38.476422    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5"]
Nov 03 00:21:38.482820 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:38.482791    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5"
Nov 03 00:21:38.502086 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:38.502043    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5"]
Nov 03 00:21:38.508031 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:38.507976    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqw59\" (UniqueName: \"kubernetes.io/projected/4162c309-506a-4cd5-92cf-769db0d6f8ef-kube-api-access-xqw59\") pod \"payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5\" (UID: \"4162c309-506a-4cd5-92cf-769db0d6f8ef\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5"
Nov 03 00:21:38.508372 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:38.508311    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4162c309-506a-4cd5-92cf-769db0d6f8ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5\" (UID: \"4162c309-506a-4cd5-92cf-769db0d6f8ef\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5"
Nov 03 00:21:38.533020 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:38.532981    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9f76b708\""
Nov 03 00:21:38.533490 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:38.533469    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9f76b708-dockercfg-4pkrc\""
Nov 03 00:21:38.639168 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:38.639128    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqw59\" (UniqueName: \"kubernetes.io/projected/4162c309-506a-4cd5-92cf-769db0d6f8ef-kube-api-access-xqw59\") pod \"payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5\" (UID: \"4162c309-506a-4cd5-92cf-769db0d6f8ef\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5"
Nov 03 00:21:38.639798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:38.639740    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4162c309-506a-4cd5-92cf-769db0d6f8ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5\" (UID: \"4162c309-506a-4cd5-92cf-769db0d6f8ef\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5"
Nov 03 00:21:38.640712 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:38.640691    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4162c309-506a-4cd5-92cf-769db0d6f8ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5\" (UID: \"4162c309-506a-4cd5-92cf-769db0d6f8ef\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5"
Nov 03 00:21:38.678979 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:38.678943    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqw59\" (UniqueName: \"kubernetes.io/projected/4162c309-506a-4cd5-92cf-769db0d6f8ef-kube-api-access-xqw59\") pod \"payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5\" (UID: \"4162c309-506a-4cd5-92cf-769db0d6f8ef\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5"
Nov 03 00:21:38.801815 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:38.801712    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5"
Nov 03 00:21:39.125625 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:21:39.125595    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4162c309_506a_4cd5_92cf_769db0d6f8ef.slice/crio-08b4ad7fc203f6c37367ae24dd4884089eb3cef6e0b0947306a6ec5b31e63244 WatchSource:0}: Error finding container 08b4ad7fc203f6c37367ae24dd4884089eb3cef6e0b0947306a6ec5b31e63244: Status 404 returned error can't find the container with id 08b4ad7fc203f6c37367ae24dd4884089eb3cef6e0b0947306a6ec5b31e63244
Nov 03 00:21:39.125781 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:39.125637    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5"]
Nov 03 00:21:40.015864 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:40.015823    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5" event={"ID":"4162c309-506a-4cd5-92cf-769db0d6f8ef","Type":"ContainerStarted","Data":"be33d071ad5dfcacc97fc0dbb6c5730fb03840d91a22ac252c9c4cbf40ddf0b6"}
Nov 03 00:21:40.016367 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:40.015958    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5"
Nov 03 00:21:40.016367 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:40.015983    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5" event={"ID":"4162c309-506a-4cd5-92cf-769db0d6f8ef","Type":"ContainerStarted","Data":"08b4ad7fc203f6c37367ae24dd4884089eb3cef6e0b0947306a6ec5b31e63244"}
Nov 03 00:21:40.050164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:40.050096    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5" podStartSLOduration=2.050082345 podStartE2EDuration="2.050082345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:40.0492313 +0000 UTC m=+982.928815342" watchObservedRunningTime="2026-11-03 00:21:40.050082345 +0000 UTC m=+982.929666387"
Nov 03 00:21:43.601055 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:43.600999    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:43.718972 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:43.718924    5012 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:43.753103 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:43.753028    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:44.039991 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:44.039813    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:44.071442 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:44.070994    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:44.246934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:44.245996    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:44.413329 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:44.412998    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:21:44.413329 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:44.413063    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx" podUID="5338078f-b28b-4f50-9594-a7a0647050a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:44.468937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:44.464035    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:21:44.468937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:44.464122    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh" podUID="ee103686-3d06-4f4c-ab26-111525f4b2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:44.584932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:44.584080    5012 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:21:44.584932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:44.584147    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb" podUID="76dcaa9e-a309-4329-8206-7d5ba46cc624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:44.606127 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:44.606083    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded"
Nov 03 00:21:44.608357 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:44.608332    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:21:44.889935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:44.880077    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 03 00:21:44.891719 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:44.890527    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:21:45.052960 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:45.049037    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:45.112940 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:45.080336    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:21:45.187019 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:45.146514    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:45.187019 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:45.146965    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:21:45.292202 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:45.291095    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:45.430938 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:45.423128    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:21:45.562140 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:45.562051    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:21:45.562406 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:45.562388    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:21:45.608755 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:45.608701    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded"
Nov 03 00:21:45.747856 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:45.747827    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:21:45.815727 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:45.815509    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:21:45.818568 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:45.818470    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:21:48.438262 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.438225    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"]
Nov 03 00:21:48.444309 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.444284    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.488989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.488951    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"]
Nov 03 00:21:48.520189 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.520149    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.520392 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.520294    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.520392 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.520344    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f130bb-d23a-4a79-b458-86a198845567-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.520516 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.520402    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j256j\" (UniqueName: \"kubernetes.io/projected/81f130bb-d23a-4a79-b458-86a198845567-kube-api-access-j256j\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.520516 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.520484    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f130bb-d23a-4a79-b458-86a198845567-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.520791 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.520539    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.520791 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.520567    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.520791 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.520606    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.520791 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.520640    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f130bb-d23a-4a79-b458-86a198845567-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.539451 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.539414    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"]
Nov 03 00:21:48.544315 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.544281    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"
Nov 03 00:21:48.601547 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.601505    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"]
Nov 03 00:21:48.623101 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.623062    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.623312 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.623115    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d989a9a9-bedb-45a3-98a7-ac5ba6929b9d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrm6m\" (UID: \"d989a9a9-bedb-45a3-98a7-ac5ba6929b9d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"
Nov 03 00:21:48.623312 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.623147    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d989a9a9-bedb-45a3-98a7-ac5ba6929b9d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrm6m\" (UID: \"d989a9a9-bedb-45a3-98a7-ac5ba6929b9d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"
Nov 03 00:21:48.623312 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.623187    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d989a9a9-bedb-45a3-98a7-ac5ba6929b9d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrm6m\" (UID: \"d989a9a9-bedb-45a3-98a7-ac5ba6929b9d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"
Nov 03 00:21:48.623312 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.623284    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f130bb-d23a-4a79-b458-86a198845567-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.623550 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.623528    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.623650 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.623634    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j256j\" (UniqueName: \"kubernetes.io/projected/81f130bb-d23a-4a79-b458-86a198845567-kube-api-access-j256j\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.623750 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.623732    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d989a9a9-bedb-45a3-98a7-ac5ba6929b9d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrm6m\" (UID: \"d989a9a9-bedb-45a3-98a7-ac5ba6929b9d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"
Nov 03 00:21:48.623852 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.623831    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2ltf\" (UniqueName: \"kubernetes.io/projected/d989a9a9-bedb-45a3-98a7-ac5ba6929b9d-kube-api-access-n2ltf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrm6m\" (UID: \"d989a9a9-bedb-45a3-98a7-ac5ba6929b9d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"
Nov 03 00:21:48.623903 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.623878    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f130bb-d23a-4a79-b458-86a198845567-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.624140 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.624124    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.624216 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.624152    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.624216 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.624183    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.624216 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.624205    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d989a9a9-bedb-45a3-98a7-ac5ba6929b9d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrm6m\" (UID: \"d989a9a9-bedb-45a3-98a7-ac5ba6929b9d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"
Nov 03 00:21:48.624324 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.624222    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d989a9a9-bedb-45a3-98a7-ac5ba6929b9d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrm6m\" (UID: \"d989a9a9-bedb-45a3-98a7-ac5ba6929b9d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"
Nov 03 00:21:48.624324 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.624248    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f130bb-d23a-4a79-b458-86a198845567-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.624394 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.624349    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.624783 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.624755    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.624886 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.624816    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.625025 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.625006    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f130bb-d23a-4a79-b458-86a198845567-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.625085 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.625021    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.627488 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.627457    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f130bb-d23a-4a79-b458-86a198845567-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.627841 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.627823    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.646472 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.646420    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 00:21:48.646472 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.646460    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 00:21:48.646703 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.646626    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 00:21:48.646746 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.646732    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wclrs\""
Nov 03 00:21:48.659470 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.659439    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j256j\" (UniqueName: \"kubernetes.io/projected/81f130bb-d23a-4a79-b458-86a198845567-kube-api-access-j256j\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.663085 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.663047    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f130bb-d23a-4a79-b458-86a198845567-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.726231 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.726128    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d989a9a9-bedb-45a3-98a7-ac5ba6929b9d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrm6m\" (UID: \"d989a9a9-bedb-45a3-98a7-ac5ba6929b9d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"
Nov 03 00:21:48.726231 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.726184    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2ltf\" (UniqueName: \"kubernetes.io/projected/d989a9a9-bedb-45a3-98a7-ac5ba6929b9d-kube-api-access-n2ltf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrm6m\" (UID: \"d989a9a9-bedb-45a3-98a7-ac5ba6929b9d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"
Nov 03 00:21:48.726488 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.726254    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d989a9a9-bedb-45a3-98a7-ac5ba6929b9d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrm6m\" (UID: \"d989a9a9-bedb-45a3-98a7-ac5ba6929b9d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"
Nov 03 00:21:48.726488 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.726270    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d989a9a9-bedb-45a3-98a7-ac5ba6929b9d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrm6m\" (UID: \"d989a9a9-bedb-45a3-98a7-ac5ba6929b9d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"
Nov 03 00:21:48.726488 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.726413    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d989a9a9-bedb-45a3-98a7-ac5ba6929b9d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrm6m\" (UID: \"d989a9a9-bedb-45a3-98a7-ac5ba6929b9d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"
Nov 03 00:21:48.726488 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.726431    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d989a9a9-bedb-45a3-98a7-ac5ba6929b9d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrm6m\" (UID: \"d989a9a9-bedb-45a3-98a7-ac5ba6929b9d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"
Nov 03 00:21:48.726488 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.726454    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d989a9a9-bedb-45a3-98a7-ac5ba6929b9d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrm6m\" (UID: \"d989a9a9-bedb-45a3-98a7-ac5ba6929b9d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"
Nov 03 00:21:48.727354 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.727326    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d989a9a9-bedb-45a3-98a7-ac5ba6929b9d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrm6m\" (UID: \"d989a9a9-bedb-45a3-98a7-ac5ba6929b9d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"
Nov 03 00:21:48.728752 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.728720    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d989a9a9-bedb-45a3-98a7-ac5ba6929b9d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrm6m\" (UID: \"d989a9a9-bedb-45a3-98a7-ac5ba6929b9d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"
Nov 03 00:21:48.728881 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.728862    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d989a9a9-bedb-45a3-98a7-ac5ba6929b9d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrm6m\" (UID: \"d989a9a9-bedb-45a3-98a7-ac5ba6929b9d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"
Nov 03 00:21:48.729206 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.729186    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d989a9a9-bedb-45a3-98a7-ac5ba6929b9d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrm6m\" (UID: \"d989a9a9-bedb-45a3-98a7-ac5ba6929b9d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"
Nov 03 00:21:48.729275 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.729213    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d989a9a9-bedb-45a3-98a7-ac5ba6929b9d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrm6m\" (UID: \"d989a9a9-bedb-45a3-98a7-ac5ba6929b9d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"
Nov 03 00:21:48.744429 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.744397    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d989a9a9-bedb-45a3-98a7-ac5ba6929b9d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrm6m\" (UID: \"d989a9a9-bedb-45a3-98a7-ac5ba6929b9d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"
Nov 03 00:21:48.744572 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.744484    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2ltf\" (UniqueName: \"kubernetes.io/projected/d989a9a9-bedb-45a3-98a7-ac5ba6929b9d-kube-api-access-n2ltf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wrm6m\" (UID: \"d989a9a9-bedb-45a3-98a7-ac5ba6929b9d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"
Nov 03 00:21:48.763185 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.763140    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:48.878257 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:48.878101    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"
Nov 03 00:21:49.088060 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:21:49.088020    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f130bb_d23a_4a79_b458_86a198845567.slice/crio-b9e5e09089ee33999f8c1b94a54606d57b6310a2584789b686c321c9bf6147d2 WatchSource:0}: Error finding container b9e5e09089ee33999f8c1b94a54606d57b6310a2584789b686c321c9bf6147d2: Status 404 returned error can't find the container with id b9e5e09089ee33999f8c1b94a54606d57b6310a2584789b686c321c9bf6147d2
Nov 03 00:21:49.088723 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:49.088690    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"]
Nov 03 00:21:49.090833 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:49.090801    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:49.090904 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:49.090868    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:49.090904 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:49.090898    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:49.198005 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:49.197748    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"]
Nov 03 00:21:49.203100 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:49.202183    5012 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:21:49.265128 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:49.265084    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m" event={"ID":"d989a9a9-bedb-45a3-98a7-ac5ba6929b9d","Type":"ContainerStarted","Data":"3556427b55a7d0681ba0d8e5b070e0e2ccfb7cef1c947b5b1c412fb6e13b46a1"}
Nov 03 00:21:49.267280 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:49.267243    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9" event={"ID":"81f130bb-d23a-4a79-b458-86a198845567","Type":"ContainerStarted","Data":"5bea062f3bbac1d808fc134907b4ea491b1005bdf592c94134baec8276833f98"}
Nov 03 00:21:49.267484 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:49.267313    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9" event={"ID":"81f130bb-d23a-4a79-b458-86a198845567","Type":"ContainerStarted","Data":"b9e5e09089ee33999f8c1b94a54606d57b6310a2584789b686c321c9bf6147d2"}
Nov 03 00:21:49.314118 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:49.314046    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9" podStartSLOduration=1.314027694 podStartE2EDuration="1.314027694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:49.308386727 +0000 UTC m=+992.187970771" watchObservedRunningTime="2026-11-03 00:21:49.314027694 +0000 UTC m=+992.193611736"
Nov 03 00:21:49.765236 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:49.765200    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:50.766030 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:50.765986    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 03 00:21:50.766528 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:50.766057    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9" podUID="81f130bb-d23a-4a79-b458-86a198845567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 03 00:21:51.029716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:51.029632    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5"
Nov 03 00:21:51.764835 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:51.764796    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 03 00:21:51.765063 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:51.764863    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9" podUID="81f130bb-d23a-4a79-b458-86a198845567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 03 00:21:52.763507 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:52.763473    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 03 00:21:52.763990 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:52.763542    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9" podUID="81f130bb-d23a-4a79-b458-86a198845567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 03 00:21:53.342756 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:53.342718    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:53.342900 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:53.342787    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:53.764085 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:53.764049    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 03 00:21:53.764531 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:53.764101    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9" podUID="81f130bb-d23a-4a79-b458-86a198845567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 03 00:21:54.349425 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:54.349383    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m" event={"ID":"d989a9a9-bedb-45a3-98a7-ac5ba6929b9d","Type":"ContainerStarted","Data":"aaf845bc3264d47d352e484b55a521b9c7d6d0ac7d728555cc0fd2990267c9da"}
Nov 03 00:21:54.351325 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:54.351249    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"
Nov 03 00:21:54.361924 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:54.361858    5012 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-wrm6m container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:21:54.362264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:54.362230    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m" podUID="d989a9a9-bedb-45a3-98a7-ac5ba6929b9d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:21:54.434063 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:54.433982    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m" podStartSLOduration=2.29517929 podStartE2EDuration="6.433962269s" totalImagesPullingTime="4.138782979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:54.430287059 +0000 UTC m=+997.309871104" watchObservedRunningTime="2026-11-03 00:21:54.433962269 +0000 UTC m=+997.313546313"
Nov 03 00:21:54.763942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:54.763830    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 03 00:21:54.763942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:54.763892    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9" podUID="81f130bb-d23a-4a79-b458-86a198845567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 03 00:21:55.360133 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:55.360098    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wrm6m"
Nov 03 00:21:55.764594 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:55.764502    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 03 00:21:55.764594 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:55.764572    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9" podUID="81f130bb-d23a-4a79-b458-86a198845567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 03 00:21:56.768235 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:56.768198    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:56.768762 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:56.768625    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:21:56.769431 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:21:56.769408    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:22:03.576041 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.575997    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"]
Nov 03 00:22:03.599324 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.599292    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:03.652421 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.646818    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"]
Nov 03 00:22:03.827987 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.827683    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:03.827987 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.827789    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:03.827987 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.827818    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:03.827987 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.827845    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:03.827987 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.827937    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:03.839322 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.828018    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:03.839322 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.828069    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:03.839322 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.828096    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:03.839322 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.828120    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx9f9\" (UniqueName: \"kubernetes.io/projected/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-kube-api-access-tx9f9\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:03.956522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.955344    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:03.956522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.955462    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:03.956522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.955521    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:03.956522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.955547    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:03.956522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.955574    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx9f9\" (UniqueName: \"kubernetes.io/projected/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-kube-api-access-tx9f9\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:03.956522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.955741    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:03.956522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.955828    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:03.956522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.955856    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:03.956522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.955882    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:03.970944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.970359    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:03.970944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.970401    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:03.974014 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.971537    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:04.046933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.982587    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:04.046933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.982829    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:04.046933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:03.983163    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:04.046933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:04.040500    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:04.076639 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:04.075790    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx9f9\" (UniqueName: \"kubernetes.io/projected/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-kube-api-access-tx9f9\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:04.076639 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:04.076438    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:04.134963 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:04.131198    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:04.620992 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:04.620954    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:22:04.621307 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:04.621024    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:22:04.621307 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:04.621054    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:22:04.622006 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:04.621977    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"]
Nov 03 00:22:05.539934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:05.539771    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd" event={"ID":"2f8b0c71-4f60-47e3-9a22-7f25e85b690c","Type":"ContainerStarted","Data":"ca6073d57aee0e6959f3d0ac717d95d3dac931794354b30bde191282a161ccda"}
Nov 03 00:22:05.539934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:05.539851    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd" event={"ID":"2f8b0c71-4f60-47e3-9a22-7f25e85b690c","Type":"ContainerStarted","Data":"1b1b8719974466e0a65f538798bfc3f1c6f31bfd24080c1c0e97bb21072c5bc0"}
Nov 03 00:22:05.612151 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:05.608748    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd" podStartSLOduration=2.608728426 podStartE2EDuration="2.608728426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:05.605596439 +0000 UTC m=+1008.485180483" watchObservedRunningTime="2026-11-03 00:22:05.608728426 +0000 UTC m=+1008.488312469"
Nov 03 00:22:06.137105 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:06.137065    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:06.139902 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:06.139594    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 03 00:22:06.139902 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:06.139681    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd" podUID="2f8b0c71-4f60-47e3-9a22-7f25e85b690c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 03 00:22:07.133804 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:07.133767    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 03 00:22:07.134126 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:07.134096    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd" podUID="2f8b0c71-4f60-47e3-9a22-7f25e85b690c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 03 00:22:07.366767 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:22:07.366728    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:22:07.552498 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:07.549004    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17eae966-8g62t"]
Nov 03 00:22:07.556688 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:07.556659    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17eae966-8g62t"]
Nov 03 00:22:07.556839 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:07.556769    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17eae966-8g62t"
Nov 03 00:22:07.566609 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:07.566580    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4896de11-cad4-4fda-a543-89c4cce5c068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17eae966-8g62t\" (UID: \"4896de11-cad4-4fda-a543-89c4cce5c068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17eae966-8g62t"
Nov 03 00:22:07.566819 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:07.566793    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm5z8\" (UniqueName: \"kubernetes.io/projected/4896de11-cad4-4fda-a543-89c4cce5c068-kube-api-access-fm5z8\") pod \"maas-api-revoke-sub-17eae966-8g62t\" (UID: \"4896de11-cad4-4fda-a543-89c4cce5c068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17eae966-8g62t"
Nov 03 00:22:07.566995 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:07.566978    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4896de11-cad4-4fda-a543-89c4cce5c068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17eae966-8g62t\" (UID: \"4896de11-cad4-4fda-a543-89c4cce5c068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17eae966-8g62t"
Nov 03 00:22:07.582187 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:07.582162    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 00:22:07.582454 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:07.582434    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 00:22:07.657519 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:07.653414    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:22:07.657519 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:07.653517    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:22:07.676073 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:07.676039    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4896de11-cad4-4fda-a543-89c4cce5c068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17eae966-8g62t\" (UID: \"4896de11-cad4-4fda-a543-89c4cce5c068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17eae966-8g62t"
Nov 03 00:22:07.676318 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:07.676296    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm5z8\" (UniqueName: \"kubernetes.io/projected/4896de11-cad4-4fda-a543-89c4cce5c068-kube-api-access-fm5z8\") pod \"maas-api-revoke-sub-17eae966-8g62t\" (UID: \"4896de11-cad4-4fda-a543-89c4cce5c068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17eae966-8g62t"
Nov 03 00:22:07.676443 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:07.676428    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4896de11-cad4-4fda-a543-89c4cce5c068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17eae966-8g62t\" (UID: \"4896de11-cad4-4fda-a543-89c4cce5c068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17eae966-8g62t"
Nov 03 00:22:07.677534 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:07.677509    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4896de11-cad4-4fda-a543-89c4cce5c068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17eae966-8g62t\" (UID: \"4896de11-cad4-4fda-a543-89c4cce5c068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17eae966-8g62t"
Nov 03 00:22:07.682298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:07.682274    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4896de11-cad4-4fda-a543-89c4cce5c068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17eae966-8g62t\" (UID: \"4896de11-cad4-4fda-a543-89c4cce5c068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17eae966-8g62t"
Nov 03 00:22:07.750896 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:07.750860    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm5z8\" (UniqueName: \"kubernetes.io/projected/4896de11-cad4-4fda-a543-89c4cce5c068-kube-api-access-fm5z8\") pod \"maas-api-revoke-sub-17eae966-8g62t\" (UID: \"4896de11-cad4-4fda-a543-89c4cce5c068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17eae966-8g62t"
Nov 03 00:22:07.954972 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:07.954650    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17eae966-8g62t"
Nov 03 00:22:08.149946 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:08.149735    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 03 00:22:08.149946 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:08.149808    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd" podUID="2f8b0c71-4f60-47e3-9a22-7f25e85b690c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 03 00:22:08.348554 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:08.348523    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17eae966-8g62t"]
Nov 03 00:22:08.583860 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:08.583785    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17eae966-8g62t" event={"ID":"4896de11-cad4-4fda-a543-89c4cce5c068","Type":"ContainerStarted","Data":"25a933557c3268077b869632367b63b21f5c9f9db43e9e8b7cb17d544d12596d"}
Nov 03 00:22:09.132768 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:09.132740    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 03 00:22:09.132971 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:09.132950    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd" podUID="2f8b0c71-4f60-47e3-9a22-7f25e85b690c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 03 00:22:09.598935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:09.598782    5012 generic.go:356] "Generic (PLEG): container finished" podID="4896de11-cad4-4fda-a543-89c4cce5c068" containerID="0f422e1940b613f10f4b17824f07b7d8dc086b7a404940bfb67715b32012d5d5" exitCode=0
Nov 03 00:22:09.598935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:09.598851    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17eae966-8g62t" event={"ID":"4896de11-cad4-4fda-a543-89c4cce5c068","Type":"ContainerDied","Data":"0f422e1940b613f10f4b17824f07b7d8dc086b7a404940bfb67715b32012d5d5"}
Nov 03 00:22:10.132070 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:10.132030    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 03 00:22:10.132307 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:10.132094    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd" podUID="2f8b0c71-4f60-47e3-9a22-7f25e85b690c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 03 00:22:10.836446 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:10.836413    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17eae966-8g62t"
Nov 03 00:22:10.844600 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:10.844574    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4896de11-cad4-4fda-a543-89c4cce5c068-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4896de11-cad4-4fda-a543-89c4cce5c068-maas-api-xks-ca\") pod \"4896de11-cad4-4fda-a543-89c4cce5c068\" (UID: \"4896de11-cad4-4fda-a543-89c4cce5c068\") "
Nov 03 00:22:10.844729 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:10.844708    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4896de11-cad4-4fda-a543-89c4cce5c068-kube-api-access-fm5z8\" (UniqueName: \"kubernetes.io/projected/4896de11-cad4-4fda-a543-89c4cce5c068-kube-api-access-fm5z8\") pod \"4896de11-cad4-4fda-a543-89c4cce5c068\" (UID: \"4896de11-cad4-4fda-a543-89c4cce5c068\") "
Nov 03 00:22:10.844792 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:10.844778    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4896de11-cad4-4fda-a543-89c4cce5c068-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4896de11-cad4-4fda-a543-89c4cce5c068-maas-api-openshift-ca\") pod \"4896de11-cad4-4fda-a543-89c4cce5c068\" (UID: \"4896de11-cad4-4fda-a543-89c4cce5c068\") "
Nov 03 00:22:10.845180 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:10.845155    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4896de11-cad4-4fda-a543-89c4cce5c068-maas-api-openshift-ca" pod "4896de11-cad4-4fda-a543-89c4cce5c068" (UID: "4896de11-cad4-4fda-a543-89c4cce5c068"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:22:10.845368 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:10.845350    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4896de11-cad4-4fda-a543-89c4cce5c068-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:22:10.846614 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:10.846590    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4896de11-cad4-4fda-a543-89c4cce5c068-maas-api-xks-ca" pod "4896de11-cad4-4fda-a543-89c4cce5c068" (UID: "4896de11-cad4-4fda-a543-89c4cce5c068"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:22:10.847028 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:10.847004    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4896de11-cad4-4fda-a543-89c4cce5c068-kube-api-access-fm5z8" pod "4896de11-cad4-4fda-a543-89c4cce5c068" (UID: "4896de11-cad4-4fda-a543-89c4cce5c068"). InnerVolumeSpecName "kube-api-access-fm5z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:22:10.947020 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:10.946856    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4896de11-cad4-4fda-a543-89c4cce5c068-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:22:10.947020 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:10.946890    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm5z8\" (UniqueName: \"kubernetes.io/projected/4896de11-cad4-4fda-a543-89c4cce5c068-kube-api-access-fm5z8\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:22:11.132991 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:11.132950    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Nov 03 00:22:11.133152 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:11.133020    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd" podUID="2f8b0c71-4f60-47e3-9a22-7f25e85b690c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Nov 03 00:22:11.630876 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:11.630843    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17eae966-8g62t"
Nov 03 00:22:11.631080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:11.630843    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17eae966-8g62t" event={"ID":"4896de11-cad4-4fda-a543-89c4cce5c068","Type":"ContainerDied","Data":"25a933557c3268077b869632367b63b21f5c9f9db43e9e8b7cb17d544d12596d"}
Nov 03 00:22:11.631080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:11.630960    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25a933557c3268077b869632367b63b21f5c9f9db43e9e8b7cb17d544d12596d"
Nov 03 00:22:12.428180 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:12.428145    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:12.428693 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:12.428635    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:12.429524 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:12.429500    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:22:13.653866 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:13.653835    5012 generic.go:356] "Generic (PLEG): container finished" podID="4a15f2a8-2b21-454f-aa1e-8cd7c2026efa" containerID="6bab4d0ce3444e7ed0526108ea75429cf427b9fff7c744793d938bf1bd2ac85c" exitCode=0
Nov 03 00:22:13.654360 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:13.653929    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5" event={"ID":"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa","Type":"ContainerDied","Data":"6bab4d0ce3444e7ed0526108ea75429cf427b9fff7c744793d938bf1bd2ac85c"}
Nov 03 00:22:13.654360 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:13.654336    5012 scope.go:122] "RemoveContainer" containerID="6bab4d0ce3444e7ed0526108ea75429cf427b9fff7c744793d938bf1bd2ac85c"
Nov 03 00:22:13.655826 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:13.655798    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:22:13.655974 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:13.655883    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:22:13.655974 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:13.655948    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:22:14.673880 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:14.673847    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5" event={"ID":"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa","Type":"ContainerStarted","Data":"0dc4924e0c292faf03cacbabb55997c51eeeec6f9ac406b65b113833029b14c2"}
Nov 03 00:22:15.577523 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:15.577485    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:22:15.578797 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:15.578769    5012 patch_prober.go:29] interesting pod/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 00:22:15.578940 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:15.578815    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5" podUID="4a15f2a8-2b21-454f-aa1e-8cd7c2026efa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 00:22:16.578104 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:16.578073    5012 patch_prober.go:29] interesting pod/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 00:22:16.578580 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:16.578138    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5" podUID="4a15f2a8-2b21-454f-aa1e-8cd7c2026efa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 00:22:17.578084 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:17.578052    5012 patch_prober.go:29] interesting pod/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 00:22:17.578266 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:17.578104    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5" podUID="4a15f2a8-2b21-454f-aa1e-8cd7c2026efa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 00:22:17.765454 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:17.765424    5012 scope.go:122] "RemoveContainer" containerID="37576e9c0a41e354c31980b24bfb4dc8f9e7614740d965916a24ee8779999cf4"
Nov 03 00:22:18.577718 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.577684    5012 patch_prober.go:29] interesting pod/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 00:22:18.577892 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.577744    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5" podUID="4a15f2a8-2b21-454f-aa1e-8cd7c2026efa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 00:22:18.585503 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.585475    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"]
Nov 03 00:22:18.589931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.586284    5012 cpu_manager.go:414] "containerMap: removing container" podUID="4896de11-cad4-4fda-a543-89c4cce5c068" containerName="revoke-keys"
Nov 03 00:22:18.589931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.586304    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4896de11-cad4-4fda-a543-89c4cce5c068" containerName="revoke-keys"
Nov 03 00:22:18.589931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.586512    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="4896de11-cad4-4fda-a543-89c4cce5c068" containerName="revoke-keys"
Nov 03 00:22:18.620226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.620195    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"]
Nov 03 00:22:18.620381 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.620297    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.711747 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.711711    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.711895 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.711783    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn8k2\" (UniqueName: \"kubernetes.io/projected/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-kube-api-access-bn8k2\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.712001 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.711977    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.712115 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.712013    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.712176 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.712158    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.712233 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.712199    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.712233 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.712220    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.712342 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.712257    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.712468 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.712449    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.813697 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.813659    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.813891 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.813709    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.813891 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.813771    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.814138 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.814112    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.814213 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.814133    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.814270 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.814233    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.814331 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.814291    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.814378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.814329    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.814459 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.814434    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.814597 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.814475    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.814597 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.814553    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn8k2\" (UniqueName: \"kubernetes.io/projected/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-kube-api-access-bn8k2\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.814726 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.814600    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.814964 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.814944    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.815182 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.815160    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.816369 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.816348    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.816923 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.816887    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.830437 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.830380    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.830830 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.830810    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn8k2\" (UniqueName: \"kubernetes.io/projected/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-kube-api-access-bn8k2\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:18.938964 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:18.938903    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:19.313952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:19.312785    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"]
Nov 03 00:22:19.313952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:19.313580    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:22:19.313952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:19.313662    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:22:19.313952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:19.313706    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:22:19.577690 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:19.577617    5012 patch_prober.go:29] interesting pod/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 00:22:19.577690 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:19.577670    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5" podUID="4a15f2a8-2b21-454f-aa1e-8cd7c2026efa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 00:22:19.743991 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:19.743944    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm" event={"ID":"a5e7d5fd-29e5-4719-8239-922e7a3aeeed","Type":"ContainerStarted","Data":"21e98aec444b140f4d9e751db08f11e48215768bba813015d7cf04cd3bce313e"}
Nov 03 00:22:19.744453 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:19.744037    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm" event={"ID":"a5e7d5fd-29e5-4719-8239-922e7a3aeeed","Type":"ContainerStarted","Data":"b6563a31625bbd1a193e2a8d9d35602969e5648bcaa2d488ca9e1bee1fa42bb6"}
Nov 03 00:22:19.788362 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:19.788297    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm" podStartSLOduration=1.788277259 podStartE2EDuration="1.788277259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:19.783374848 +0000 UTC m=+1022.662958895" watchObservedRunningTime="2026-11-03 00:22:19.788277259 +0000 UTC m=+1022.667861303"
Nov 03 00:22:19.944957 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:19.942021    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:19.973355 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:19.973307    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:22:20.581815 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:20.581786    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:22:20.754291 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:20.754262    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:22:20.941271 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:20.941238    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 03 00:22:20.941441 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:20.941286    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm" podUID="a5e7d5fd-29e5-4719-8239-922e7a3aeeed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 03 00:22:21.940017 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:21.939981    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 03 00:22:21.940422 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:21.940035    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm" podUID="a5e7d5fd-29e5-4719-8239-922e7a3aeeed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 03 00:22:22.939639 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:22.939604    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 03 00:22:22.939814 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:22.939664    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm" podUID="a5e7d5fd-29e5-4719-8239-922e7a3aeeed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 03 00:22:23.939845 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:23.939812    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 03 00:22:23.940278 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:23.939867    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm" podUID="a5e7d5fd-29e5-4719-8239-922e7a3aeeed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 03 00:22:24.942862 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:24.940262    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 03 00:22:24.942862 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:24.940325    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm" podUID="a5e7d5fd-29e5-4719-8239-922e7a3aeeed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 03 00:22:25.939701 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:25.939666    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 03 00:22:25.939878 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:25.939724    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm" podUID="a5e7d5fd-29e5-4719-8239-922e7a3aeeed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 03 00:22:26.944597 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:26.944566    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:26.945095 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:26.945074    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:26.945823 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:26.945806    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 00:22:28.576138 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:28.576029    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:28.619069 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:28.618993    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:28.751081 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:28.751036    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:28.839215 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:28.839123    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:28.997193 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:28.997149    5012 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:29.293496 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:29.293071    5012 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded"
Nov 03 00:22:29.512330 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:29.512286    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Nov 03 00:22:29.579201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:29.579098    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 03 00:22:29.581121 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:29.579939    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:22:29.626136 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:29.626094    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:29.627026 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:29.627000    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:22:29.776935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:29.760039    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:29.776935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:29.760774    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:22:29.882931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:29.879061    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:29.882931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:29.879698    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:22:30.195993 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:30.178509    5012 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:30.516984 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:30.516011    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:30.516984 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:30.516761    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:22:30.593286 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:30.592033    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:30.681942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:30.668138    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded"
Nov 03 00:22:30.776005 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:30.771204    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:30.995260 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:30.995219    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 03 00:22:31.186554 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:31.186518    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:22:31.668050 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:31.665001    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:31.718933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:31.704061    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:31.786175 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:31.786104    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 03 00:22:32.035446 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:32.033990    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 03 00:22:32.097378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:32.097345    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:22:32.669008 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:32.668012    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 03 00:22:32.706932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:32.706003    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded"
Nov 03 00:22:32.830929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:32.830839    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:32.959981 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:32.958961    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:22:33.078397 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:33.077335    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:22:33.179366 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:33.179328    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:22:37.364887 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:22:37.364846    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:22:48.649253 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.649223    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"]
Nov 03 00:22:48.654684 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.654654    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.690604 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.690576    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.691082 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.690957    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.691535 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.691421    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.691719 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.691696    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.692136 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.692030    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8760ca6d-230b-4d95-ae34-d9384b441607-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.692421 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.692315    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62m2j\" (UniqueName: \"kubernetes.io/projected/8760ca6d-230b-4d95-ae34-d9384b441607-kube-api-access-62m2j\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.692795 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.692777    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8760ca6d-230b-4d95-ae34-d9384b441607-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.693055 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.693032    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8760ca6d-230b-4d95-ae34-d9384b441607-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.693308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.693203    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.702741 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.702719    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"]
Nov 03 00:22:48.794966 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.794895    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.795149 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.795014    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.795149 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.795075    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8760ca6d-230b-4d95-ae34-d9384b441607-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.795149 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.795139    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62m2j\" (UniqueName: \"kubernetes.io/projected/8760ca6d-230b-4d95-ae34-d9384b441607-kube-api-access-62m2j\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.795335 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.795234    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8760ca6d-230b-4d95-ae34-d9384b441607-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.795335 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.795267    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8760ca6d-230b-4d95-ae34-d9384b441607-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.795335 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.795266    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.795508 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.795343    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.795508 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.795374    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.795508 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.795382    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.795508 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.795396    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.795724 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.795566    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.795825 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.795805    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.796545 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.796506    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8760ca6d-230b-4d95-ae34-d9384b441607-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.798367 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.798344    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.798657 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.798635    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8760ca6d-230b-4d95-ae34-d9384b441607-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.827972 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.821725    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62m2j\" (UniqueName: \"kubernetes.io/projected/8760ca6d-230b-4d95-ae34-d9384b441607-kube-api-access-62m2j\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.827972 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.822292    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8760ca6d-230b-4d95-ae34-d9384b441607-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:48.974189 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:48.974111    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:49.372054 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:49.372025    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"]
Nov 03 00:22:49.379859 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:22:49.379833    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8760ca6d_230b_4d95_ae34_d9384b441607.slice/crio-ae14250ddc1a20ec61cf9413022fa7506207fd5e61ac18937bb52b642697351d WatchSource:0}: Error finding container ae14250ddc1a20ec61cf9413022fa7506207fd5e61ac18937bb52b642697351d: Status 404 returned error can't find the container with id ae14250ddc1a20ec61cf9413022fa7506207fd5e61ac18937bb52b642697351d
Nov 03 00:22:49.382220 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:49.382188    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:22:49.382312 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:49.382251    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:22:49.382312 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:49.382279    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:22:50.321393 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:50.321350    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd" event={"ID":"8760ca6d-230b-4d95-ae34-d9384b441607","Type":"ContainerStarted","Data":"193de80f706d976677f9e6fe90c33184d014cb5dbe1f3a6691ffa63646e3da05"}
Nov 03 00:22:50.322004 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:50.321407    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd" event={"ID":"8760ca6d-230b-4d95-ae34-d9384b441607","Type":"ContainerStarted","Data":"ae14250ddc1a20ec61cf9413022fa7506207fd5e61ac18937bb52b642697351d"}
Nov 03 00:22:50.377657 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:50.377605    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd" podStartSLOduration=2.377592133 podStartE2EDuration="2.377592133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:50.36607895 +0000 UTC m=+1053.245662992" watchObservedRunningTime="2026-11-03 00:22:50.377592133 +0000 UTC m=+1053.257176175"
Nov 03 00:22:50.974516 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:50.974484    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:50.976178 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:50.976148    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 03 00:22:50.976301 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:50.976202    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd" podUID="8760ca6d-230b-4d95-ae34-d9384b441607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 03 00:22:51.975301 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:51.975268    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 03 00:22:51.975700 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:51.975343    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd" podUID="8760ca6d-230b-4d95-ae34-d9384b441607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 03 00:22:52.975135 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:52.975101    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 03 00:22:52.975318 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:52.975150    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd" podUID="8760ca6d-230b-4d95-ae34-d9384b441607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 03 00:22:53.974748 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:53.974716    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 03 00:22:53.974976 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:53.974773    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd" podUID="8760ca6d-230b-4d95-ae34-d9384b441607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 03 00:22:54.974756 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:54.974724    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 03 00:22:54.975166 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:54.974783    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd" podUID="8760ca6d-230b-4d95-ae34-d9384b441607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 03 00:22:55.975329 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:55.975286    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 03 00:22:55.975838 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:55.975361    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd" podUID="8760ca6d-230b-4d95-ae34-d9384b441607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 03 00:22:56.979118 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:56.979090    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:56.979527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:56.979360    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:22:56.980130 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:22:56.980109    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:23:01.047779 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:01.047711    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ae896ef-r56pz"]
Nov 03 00:23:01.057656 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:01.057613    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ae896ef-r56pz"]
Nov 03 00:23:01.245425 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:01.245391    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1" path="/var/lib/kubelet/pods/1eeebb6b-69fe-4eb9-b182-c8a6f12c05f1/volumes"
Nov 03 00:23:07.365355 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:23:07.365320    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:23:08.027757 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:08.027724    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98453474-stnhm"]
Nov 03 00:23:08.037499 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:08.037472    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98453474-stnhm"]
Nov 03 00:23:09.247530 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:09.247497    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e714a7c2-61b0-4419-a5bb-f09fb5336643" path="/var/lib/kubelet/pods/e714a7c2-61b0-4419-a5bb-f09fb5336643/volumes"
Nov 03 00:23:13.027421 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:13.027386    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ccd8e17-qvkk5"]
Nov 03 00:23:13.037841 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:13.037806    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ccd8e17-qvkk5"]
Nov 03 00:23:13.245339 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:13.245304    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca01721d-5c8f-4127-977f-5fa5eaf3100e" path="/var/lib/kubelet/pods/ca01721d-5c8f-4127-977f-5fa5eaf3100e/volumes"
Nov 03 00:23:19.031628 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:19.031594    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16172f30-lp8gm"]
Nov 03 00:23:19.039092 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:19.039065    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16172f30-lp8gm"]
Nov 03 00:23:19.246897 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:19.246854    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad67a5bf-0df3-4d50-ad9d-52f2cc529566" path="/var/lib/kubelet/pods/ad67a5bf-0df3-4d50-ad9d-52f2cc529566/volumes"
Nov 03 00:23:27.028223 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:27.028190    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6c77ced-b5sfz"]
Nov 03 00:23:27.037995 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:27.037966    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6c77ced-b5sfz"]
Nov 03 00:23:27.245155 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:27.245124    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3251a5de-51bf-4066-804f-f028a0109834" path="/var/lib/kubelet/pods/3251a5de-51bf-4066-804f-f028a0109834/volumes"
Nov 03 00:23:29.590095 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:29.590060    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5"]
Nov 03 00:23:34.591169 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:34.591113    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5" podUID="4162c309-506a-4cd5-92cf-769db0d6f8ef" containerName="payload-pre-processing" containerID="cri-o://be33d071ad5dfcacc97fc0dbb6c5730fb03840d91a22ac252c9c4cbf40ddf0b6" gracePeriod=25
Nov 03 00:23:37.364712 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:23:37.364675    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:23:41.027077 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.027037    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5" podUID="4162c309-506a-4cd5-92cf-769db0d6f8ef" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:23:41.235381 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.235337    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"]
Nov 03 00:23:41.240042 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.240010    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.256445 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.256416    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5c5686-openshift-default-dockercfg-l6rml\""
Nov 03 00:23:41.272988 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.272963    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"]
Nov 03 00:23:41.323816 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.323730    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-podinfo\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.323816 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.323796    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbcrg\" (UniqueName: \"kubernetes.io/projected/fc06ba19-a467-4dc0-8d91-b272b7151b20-kube-api-access-zbcrg\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.324004 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.323943    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-token\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.324078 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.324058    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-credential-socket\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.324174 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.324162    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-envoy\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.324210 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.324188    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc06ba19-a467-4dc0-8d91-b272b7151b20-istiod-ca-cert\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.324259 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.324222    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-data\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.324403 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.324382    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-workload-socket\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.324466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.324416    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-workload-certs\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.426659 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.426615    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-envoy\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.426659 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.426676    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc06ba19-a467-4dc0-8d91-b272b7151b20-istiod-ca-cert\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.427025 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.426743    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-data\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.427025 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.426863    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-workload-socket\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.427025 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.426888    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-workload-certs\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.427025 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.426950    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-podinfo\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.427025 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.427009    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbcrg\" (UniqueName: \"kubernetes.io/projected/fc06ba19-a467-4dc0-8d91-b272b7151b20-kube-api-access-zbcrg\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.427282 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.427068    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-token\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.427282 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.427164    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-credential-socket\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.429972 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.427643    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-credential-socket\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.429972 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.427740    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-workload-certs\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.429972 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.428489    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc06ba19-a467-4dc0-8d91-b272b7151b20-istiod-ca-cert\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.429972 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.428591    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-data\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.429972 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.428736    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-workload-socket\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.471827 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.470275    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-envoy\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.487931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.485863    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-token\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.488505 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.488474    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-podinfo\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.498087 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.498030    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbcrg\" (UniqueName: \"kubernetes.io/projected/fc06ba19-a467-4dc0-8d91-b272b7151b20-kube-api-access-zbcrg\") pod \"e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") " pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.569934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.566207    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:41.953787 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.953758    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"]
Nov 03 00:23:41.955975 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:23:41.955949    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc06ba19_a467_4dc0_8d91_b272b7151b20.slice/crio-2dc1f23bb30526f91898da21ebe5aa68ec1e9bfe5469ac41c50a2987ff6e0bd9 WatchSource:0}: Error finding container 2dc1f23bb30526f91898da21ebe5aa68ec1e9bfe5469ac41c50a2987ff6e0bd9: Status 404 returned error can't find the container with id 2dc1f23bb30526f91898da21ebe5aa68ec1e9bfe5469ac41c50a2987ff6e0bd9
Nov 03 00:23:41.958374 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.958338    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:23:41.958455 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.958404    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:23:41.958455 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:41.958432    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:23:42.862977 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:42.862941    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57" event={"ID":"fc06ba19-a467-4dc0-8d91-b272b7151b20","Type":"ContainerStarted","Data":"3937c38e9ae7805dd91d332da72f7a87413f6d7a79b53ba2b36e806ec73176cf"}
Nov 03 00:23:42.863370 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:42.862997    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57" event={"ID":"fc06ba19-a467-4dc0-8d91-b272b7151b20","Type":"ContainerStarted","Data":"2dc1f23bb30526f91898da21ebe5aa68ec1e9bfe5469ac41c50a2987ff6e0bd9"}
Nov 03 00:23:42.914054 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:42.914004    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57" podStartSLOduration=1.913991024 podStartE2EDuration="1.913991024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:42.908998595 +0000 UTC m=+1105.788582638" watchObservedRunningTime="2026-11-03 00:23:42.913991024 +0000 UTC m=+1105.793575067"
Nov 03 00:23:43.566985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:43.566956    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:43.572629 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:43.572598    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:43.871774 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:43.871744    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:43.872674 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:43.872652    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:23:47.288286 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:47.288250    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb"]
Nov 03 00:23:47.292805 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:47.292786    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb"
Nov 03 00:23:47.312840 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:47.312812    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb"]
Nov 03 00:23:47.316414 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:47.316368    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5c5686\""
Nov 03 00:23:47.316544 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:47.316525    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5c5686-dockercfg-4qgmh\""
Nov 03 00:23:47.408037 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:47.408008    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c788d\" (UniqueName: \"kubernetes.io/projected/ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782-kube-api-access-c788d\") pod \"payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb\" (UID: \"ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb"
Nov 03 00:23:47.408165 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:47.408138    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb\" (UID: \"ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb"
Nov 03 00:23:47.419804 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:47.419780    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l"]
Nov 03 00:23:47.424187 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:47.424169    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l"
Nov 03 00:23:47.443006 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:47.442981    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l"]
Nov 03 00:23:47.512932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:47.510070    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l\" (UID: \"fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1\") " pod="openshift-ingress/payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l"
Nov 03 00:23:47.512932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:47.510144    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb\" (UID: \"ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb"
Nov 03 00:23:47.512932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:47.510289    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vns4\" (UniqueName: \"kubernetes.io/projected/fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1-kube-api-access-5vns4\") pod \"payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l\" (UID: \"fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1\") " pod="openshift-ingress/payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l"
Nov 03 00:23:47.512932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:47.510531    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c788d\" (UniqueName: \"kubernetes.io/projected/ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782-kube-api-access-c788d\") pod \"payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb\" (UID: \"ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb"
Nov 03 00:23:47.512932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:47.511891    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb\" (UID: \"ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb"
Nov 03 00:23:47.541702 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:47.538695    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c788d\" (UniqueName: \"kubernetes.io/projected/ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782-kube-api-access-c788d\") pod \"payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb\" (UID: \"ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb"
Nov 03 00:23:47.608399 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:47.608371    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb"
Nov 03 00:23:47.612285 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:47.611891    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vns4\" (UniqueName: \"kubernetes.io/projected/fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1-kube-api-access-5vns4\") pod \"payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l\" (UID: \"fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1\") " pod="openshift-ingress/payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l"
Nov 03 00:23:47.612285 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:47.612212    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l\" (UID: \"fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1\") " pod="openshift-ingress/payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l"
Nov 03 00:23:47.613714 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:47.613684    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l\" (UID: \"fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1\") " pod="openshift-ingress/payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l"
Nov 03 00:23:47.640268 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:47.640238    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vns4\" (UniqueName: \"kubernetes.io/projected/fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1-kube-api-access-5vns4\") pod \"payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l\" (UID: \"fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1\") " pod="openshift-ingress/payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l"
Nov 03 00:23:47.739196 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:47.738595    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l"
Nov 03 00:23:47.975125 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:23:47.975096    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac9a2bf0_8cd4_48ec_bdf5_b1b8417c8782.slice/crio-086ef23eb59353228dec1ee6649d2276326a3e75f4f7aa82214b06d4cb66a80c WatchSource:0}: Error finding container 086ef23eb59353228dec1ee6649d2276326a3e75f4f7aa82214b06d4cb66a80c: Status 404 returned error can't find the container with id 086ef23eb59353228dec1ee6649d2276326a3e75f4f7aa82214b06d4cb66a80c
Nov 03 00:23:47.976185 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:47.976163    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb"]
Nov 03 00:23:48.077253 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:23:48.077224    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfccdbbc0_6cb5_4732_8d45_d2c8cfbf9ff1.slice/crio-2680aed1fcead60f55224428ce7329926f1b3b827c598ab9f905901e0bd1d186 WatchSource:0}: Error finding container 2680aed1fcead60f55224428ce7329926f1b3b827c598ab9f905901e0bd1d186: Status 404 returned error can't find the container with id 2680aed1fcead60f55224428ce7329926f1b3b827c598ab9f905901e0bd1d186
Nov 03 00:23:48.078168 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:48.078146    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l"]
Nov 03 00:23:48.923538 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:48.923499    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb" event={"ID":"ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782","Type":"ContainerStarted","Data":"09c4b16907633fa28ff17db1a593b135003a8502b0a4775422c74d2037603577"}
Nov 03 00:23:48.923964 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:48.923594    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb" event={"ID":"ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782","Type":"ContainerStarted","Data":"086ef23eb59353228dec1ee6649d2276326a3e75f4f7aa82214b06d4cb66a80c"}
Nov 03 00:23:48.923964 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:48.923627    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb"
Nov 03 00:23:48.925361 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:48.925338    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l" event={"ID":"fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1","Type":"ContainerStarted","Data":"5847cfb1e2820d21607236953a82cb6d439afeab62159a469346d59aa74fcec9"}
Nov 03 00:23:48.925503 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:48.925377    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l" event={"ID":"fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1","Type":"ContainerStarted","Data":"2680aed1fcead60f55224428ce7329926f1b3b827c598ab9f905901e0bd1d186"}
Nov 03 00:23:48.925503 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:48.925397    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l"
Nov 03 00:23:48.961253 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:48.961199    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb" podStartSLOduration=1.9611871330000001 podStartE2EDuration="1.961187133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:48.957691167 +0000 UTC m=+1111.837275209" watchObservedRunningTime="2026-11-03 00:23:48.961187133 +0000 UTC m=+1111.840771175"
Nov 03 00:23:48.997363 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:48.997319    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l" podStartSLOduration=1.997307441 podStartE2EDuration="1.997307441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:48.991902892 +0000 UTC m=+1111.871486934" watchObservedRunningTime="2026-11-03 00:23:48.997307441 +0000 UTC m=+1111.876891483"
Nov 03 00:23:51.027436 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:51.027399    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5" podUID="4162c309-506a-4cd5-92cf-769db0d6f8ef" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:23:54.873261 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:54.873234    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5"
Nov 03 00:23:54.945443 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:54.945415    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4162c309-506a-4cd5-92cf-769db0d6f8ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4162c309-506a-4cd5-92cf-769db0d6f8ef-plugins-config-volume\") pod \"4162c309-506a-4cd5-92cf-769db0d6f8ef\" (UID: \"4162c309-506a-4cd5-92cf-769db0d6f8ef\") "
Nov 03 00:23:54.945709 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:54.945691    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4162c309-506a-4cd5-92cf-769db0d6f8ef-kube-api-access-xqw59\" (UniqueName: \"kubernetes.io/projected/4162c309-506a-4cd5-92cf-769db0d6f8ef-kube-api-access-xqw59\") pod \"4162c309-506a-4cd5-92cf-769db0d6f8ef\" (UID: \"4162c309-506a-4cd5-92cf-769db0d6f8ef\") "
Nov 03 00:23:54.945902 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:54.945873    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4162c309-506a-4cd5-92cf-769db0d6f8ef-plugins-config-volume" pod "4162c309-506a-4cd5-92cf-769db0d6f8ef" (UID: "4162c309-506a-4cd5-92cf-769db0d6f8ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:23:54.946329 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:54.946307    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4162c309-506a-4cd5-92cf-769db0d6f8ef-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:23:54.947769 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:54.947746    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4162c309-506a-4cd5-92cf-769db0d6f8ef-kube-api-access-xqw59" pod "4162c309-506a-4cd5-92cf-769db0d6f8ef" (UID: "4162c309-506a-4cd5-92cf-769db0d6f8ef"). InnerVolumeSpecName "kube-api-access-xqw59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:23:54.996896 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:54.996869    5012 generic.go:356] "Generic (PLEG): container finished" podID="4162c309-506a-4cd5-92cf-769db0d6f8ef" containerID="be33d071ad5dfcacc97fc0dbb6c5730fb03840d91a22ac252c9c4cbf40ddf0b6" exitCode=0
Nov 03 00:23:54.997015 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:54.996947    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5"
Nov 03 00:23:54.997015 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:54.996957    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5" event={"ID":"4162c309-506a-4cd5-92cf-769db0d6f8ef","Type":"ContainerDied","Data":"be33d071ad5dfcacc97fc0dbb6c5730fb03840d91a22ac252c9c4cbf40ddf0b6"}
Nov 03 00:23:54.997084 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:54.997013    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5" event={"ID":"4162c309-506a-4cd5-92cf-769db0d6f8ef","Type":"ContainerDied","Data":"08b4ad7fc203f6c37367ae24dd4884089eb3cef6e0b0947306a6ec5b31e63244"}
Nov 03 00:23:54.997084 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:54.997037    5012 scope.go:122] "RemoveContainer" containerID="be33d071ad5dfcacc97fc0dbb6c5730fb03840d91a22ac252c9c4cbf40ddf0b6"
Nov 03 00:23:55.008761 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:55.008740    5012 scope.go:122] "RemoveContainer" containerID="be33d071ad5dfcacc97fc0dbb6c5730fb03840d91a22ac252c9c4cbf40ddf0b6"
Nov 03 00:23:55.009041 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:23:55.009023    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be33d071ad5dfcacc97fc0dbb6c5730fb03840d91a22ac252c9c4cbf40ddf0b6\": container with ID starting with be33d071ad5dfcacc97fc0dbb6c5730fb03840d91a22ac252c9c4cbf40ddf0b6 not found: ID does not exist" containerID="be33d071ad5dfcacc97fc0dbb6c5730fb03840d91a22ac252c9c4cbf40ddf0b6"
Nov 03 00:23:55.009103 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:55.009048    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be33d071ad5dfcacc97fc0dbb6c5730fb03840d91a22ac252c9c4cbf40ddf0b6"} err="failed to get container status \"be33d071ad5dfcacc97fc0dbb6c5730fb03840d91a22ac252c9c4cbf40ddf0b6\": rpc error: code = NotFound desc = could not find container \"be33d071ad5dfcacc97fc0dbb6c5730fb03840d91a22ac252c9c4cbf40ddf0b6\": container with ID starting with be33d071ad5dfcacc97fc0dbb6c5730fb03840d91a22ac252c9c4cbf40ddf0b6 not found: ID does not exist"
Nov 03 00:23:55.044139 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:55.044118    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5"]
Nov 03 00:23:55.047623 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:55.047601    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqw59\" (UniqueName: \"kubernetes.io/projected/4162c309-506a-4cd5-92cf-769db0d6f8ef-kube-api-access-xqw59\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:23:55.048831 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:55.048814    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-xfdm5"]
Nov 03 00:23:55.245387 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:55.245327    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4162c309-506a-4cd5-92cf-769db0d6f8ef" path="/var/lib/kubelet/pods/4162c309-506a-4cd5-92cf-769db0d6f8ef/volumes"
Nov 03 00:23:59.936792 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:59.936718    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb"
Nov 03 00:23:59.937184 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:23:59.936799    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l"
Nov 03 00:24:07.365073 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:24:07.365034    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:24:17.840525 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:17.840484    5012 scope.go:122] "RemoveContainer" containerID="bc4634160c87716f50e46ae36505894c76b29e0f60a0c8d0c455f596b2084d54"
Nov 03 00:24:17.854039 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:17.854011    5012 scope.go:122] "RemoveContainer" containerID="73b444ceeb3942b6d029d688e8d2dade204984165456588a9316b03d19f160ed"
Nov 03 00:24:17.865738 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:17.865715    5012 scope.go:122] "RemoveContainer" containerID="03b138aca35431f3563693f27ad115d78141f8d271e244f5040e462bd15670eb"
Nov 03 00:24:17.887405 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:17.887377    5012 scope.go:122] "RemoveContainer" containerID="f701c34fee22053aacea1b673def012fd5c70199cd5b0aff016af302e9daba69"
Nov 03 00:24:21.673933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:21.645469    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 03 00:24:21.944983 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:21.944578    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 03 00:24:22.071934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:22.071671    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded"
Nov 03 00:24:22.155578 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:22.155531    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:24:22.263992 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:22.263467    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Nov 03 00:24:22.279927 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:22.273968    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-5lnf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:24:22.279927 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:22.274029    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2" podUID="aa430510-e80c-4743-b4f3-838ce423e442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:24:22.700019 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:22.699972    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 03 00:24:22.701061 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:22.701038    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:24:23.100146 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:23.100088    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:24:23.101097 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:23.101073    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:24:23.317936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:23.306255    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Nov 03 00:24:23.330992 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:23.330893    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:24:23.701020 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:23.700979    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:24:23.717143 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:23.717086    5012 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:24:23.736243 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:23.735677    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:24:23.777932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:23.774333    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:24:23.903851 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:23.894098    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:24:29.921040 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:29.921002    5012 patch_prober.go:29] interesting pod/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:24:29.921496 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:29.921067    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57" podUID="fc06ba19-a467-4dc0-8d91-b272b7151b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:24:30.848418 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:30.848385    5012 generic.go:356] "Generic (PLEG): container finished" podID="fc06ba19-a467-4dc0-8d91-b272b7151b20" containerID="3937c38e9ae7805dd91d332da72f7a87413f6d7a79b53ba2b36e806ec73176cf" exitCode=0
Nov 03 00:24:30.848630 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:30.848465    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57" event={"ID":"fc06ba19-a467-4dc0-8d91-b272b7151b20","Type":"ContainerDied","Data":"3937c38e9ae7805dd91d332da72f7a87413f6d7a79b53ba2b36e806ec73176cf"}
Nov 03 00:24:30.848918 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:30.848892    5012 scope.go:122] "RemoveContainer" containerID="3937c38e9ae7805dd91d332da72f7a87413f6d7a79b53ba2b36e806ec73176cf"
Nov 03 00:24:30.850435 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:30.850410    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:24:30.850515 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:30.850476    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:24:30.850562 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:30.850520    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:24:31.861417 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:31.861380    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57" event={"ID":"fc06ba19-a467-4dc0-8d91-b272b7151b20","Type":"ContainerStarted","Data":"a2dc18cf2d5d516449009f80b5c1d95508169d565890ec252389f23f99af8b75"}
Nov 03 00:24:32.566925 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:32.566890    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:24:32.567124 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:32.567090    5012 patch_prober.go:29] interesting pod/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 03 00:24:32.567207 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:32.567156    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57" podUID="fc06ba19-a467-4dc0-8d91-b272b7151b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 03 00:24:33.567467 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:33.567428    5012 patch_prober.go:29] interesting pod/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 03 00:24:33.567877 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:33.567491    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57" podUID="fc06ba19-a467-4dc0-8d91-b272b7151b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 03 00:24:34.567290 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:34.567254    5012 patch_prober.go:29] interesting pod/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 03 00:24:34.567457 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:34.567315    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57" podUID="fc06ba19-a467-4dc0-8d91-b272b7151b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 03 00:24:35.567175 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:35.567139    5012 patch_prober.go:29] interesting pod/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 03 00:24:35.567568 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:35.567201    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57" podUID="fc06ba19-a467-4dc0-8d91-b272b7151b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 03 00:24:36.567934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:36.567871    5012 patch_prober.go:29] interesting pod/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 03 00:24:36.568394 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:36.567947    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57" podUID="fc06ba19-a467-4dc0-8d91-b272b7151b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 03 00:24:37.367001 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:24:37.366965    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:24:37.571547 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:37.571516    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:24:37.572051 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:37.572032    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:24:37.572802 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:37.572783    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:24:48.350966 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:48.350932    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ef32911-k2gjm"]
Nov 03 00:24:48.351796 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:48.351773    5012 cpu_manager.go:414] "containerMap: removing container" podUID="4162c309-506a-4cd5-92cf-769db0d6f8ef" containerName="payload-pre-processing"
Nov 03 00:24:48.351898 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:48.351796    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4162c309-506a-4cd5-92cf-769db0d6f8ef" containerName="payload-pre-processing"
Nov 03 00:24:48.352040 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:48.352025    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="4162c309-506a-4cd5-92cf-769db0d6f8ef" containerName="payload-pre-processing"
Nov 03 00:24:48.356222 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:48.356201    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef32911-k2gjm"
Nov 03 00:24:48.375069 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:48.375044    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ef32911-k2gjm"]
Nov 03 00:24:48.379278 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:48.379248    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:24:48.379415 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:48.379286    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:24:48.420427 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:48.416167    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 00:24:48.420427 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:48.416248    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 00:24:48.426621 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:48.426589    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ef32911-k2gjm\" (UID: \"3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef32911-k2gjm"
Nov 03 00:24:48.426758 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:48.426710    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ef32911-k2gjm\" (UID: \"3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef32911-k2gjm"
Nov 03 00:24:48.426888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:48.426870    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw9bv\" (UniqueName: \"kubernetes.io/projected/3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed-kube-api-access-tw9bv\") pod \"maas-api-revoke-sub-3ef32911-k2gjm\" (UID: \"3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef32911-k2gjm"
Nov 03 00:24:48.528234 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:48.528197    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ef32911-k2gjm\" (UID: \"3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef32911-k2gjm"
Nov 03 00:24:48.528495 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:48.528294    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ef32911-k2gjm\" (UID: \"3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef32911-k2gjm"
Nov 03 00:24:48.528495 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:48.528441    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw9bv\" (UniqueName: \"kubernetes.io/projected/3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed-kube-api-access-tw9bv\") pod \"maas-api-revoke-sub-3ef32911-k2gjm\" (UID: \"3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef32911-k2gjm"
Nov 03 00:24:48.529122 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:48.529089    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ef32911-k2gjm\" (UID: \"3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef32911-k2gjm"
Nov 03 00:24:48.530668 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:48.530640    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ef32911-k2gjm\" (UID: \"3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef32911-k2gjm"
Nov 03 00:24:48.547381 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:48.547355    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw9bv\" (UniqueName: \"kubernetes.io/projected/3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed-kube-api-access-tw9bv\") pod \"maas-api-revoke-sub-3ef32911-k2gjm\" (UID: \"3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef32911-k2gjm"
Nov 03 00:24:48.672966 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:48.672887    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef32911-k2gjm"
Nov 03 00:24:48.989181 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:48.989107    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ef32911-k2gjm"]
Nov 03 00:24:48.993965 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:24:48.993899    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c5bb4d3_7aa8_4213_be5c_2cf52eecbfed.slice/crio-3484a213815f3742cf15d03694d2f42832e025c0f03068187fce8c37f52aaa53 WatchSource:0}: Error finding container 3484a213815f3742cf15d03694d2f42832e025c0f03068187fce8c37f52aaa53: Status 404 returned error can't find the container with id 3484a213815f3742cf15d03694d2f42832e025c0f03068187fce8c37f52aaa53
Nov 03 00:24:49.123357 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:49.123325    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef32911-k2gjm" event={"ID":"3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed","Type":"ContainerStarted","Data":"3484a213815f3742cf15d03694d2f42832e025c0f03068187fce8c37f52aaa53"}
Nov 03 00:24:50.133398 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:50.133364    5012 generic.go:356] "Generic (PLEG): container finished" podID="3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed" containerID="bebfb19ca5975e6198ca5e0f922d2342996098e8c36802fe3a67a08d65d0d698" exitCode=0
Nov 03 00:24:50.133771 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:50.133437    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef32911-k2gjm" event={"ID":"3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed","Type":"ContainerDied","Data":"bebfb19ca5975e6198ca5e0f922d2342996098e8c36802fe3a67a08d65d0d698"}
Nov 03 00:24:51.320828 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:51.320804    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef32911-k2gjm"
Nov 03 00:24:51.369546 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:51.369513    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed-maas-api-openshift-ca\") pod \"3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed\" (UID: \"3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed\") "
Nov 03 00:24:51.369698 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:51.369570    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed-kube-api-access-tw9bv\" (UniqueName: \"kubernetes.io/projected/3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed-kube-api-access-tw9bv\") pod \"3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed\" (UID: \"3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed\") "
Nov 03 00:24:51.369698 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:51.369652    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed-maas-api-xks-ca\") pod \"3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed\" (UID: \"3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed\") "
Nov 03 00:24:51.370004 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:51.369968    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed-maas-api-openshift-ca" pod "3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed" (UID: "3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:24:51.370659 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:51.370632    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:24:51.373370 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:51.373340    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed-maas-api-xks-ca" pod "3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed" (UID: "3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:24:51.373480 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:51.373467    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed-kube-api-access-tw9bv" pod "3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed" (UID: "3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed"). InnerVolumeSpecName "kube-api-access-tw9bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:24:51.475427 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:51.475225    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw9bv\" (UniqueName: \"kubernetes.io/projected/3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed-kube-api-access-tw9bv\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:24:51.475427 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:51.475328    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:24:52.150574 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:52.150542    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef32911-k2gjm"
Nov 03 00:24:52.150574 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:52.150561    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef32911-k2gjm" event={"ID":"3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed","Type":"ContainerDied","Data":"3484a213815f3742cf15d03694d2f42832e025c0f03068187fce8c37f52aaa53"}
Nov 03 00:24:52.150808 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:24:52.150608    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3484a213815f3742cf15d03694d2f42832e025c0f03068187fce8c37f52aaa53"
Nov 03 00:25:02.027227 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:02.027192    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-885c6809-mms4m"]
Nov 03 00:25:02.037479 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:02.037452    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-885c6809-mms4m"]
Nov 03 00:25:03.246012 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:03.245980    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="354fe5c6-af53-4068-8cf4-48d8fa63c611" path="/var/lib/kubelet/pods/354fe5c6-af53-4068-8cf4-48d8fa63c611/volumes"
Nov 03 00:25:05.682252 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:05.682219    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm"]
Nov 03 00:25:05.683167 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:05.683148    5012 cpu_manager.go:414] "containerMap: removing container" podUID="3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed" containerName="revoke-keys"
Nov 03 00:25:05.683219 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:05.683169    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed" containerName="revoke-keys"
Nov 03 00:25:05.683284 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:05.683275    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed" containerName="revoke-keys"
Nov 03 00:25:05.686841 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:05.686824    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm"
Nov 03 00:25:05.704032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:05.703997    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm"]
Nov 03 00:25:05.706619 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:05.706593    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f3806a\""
Nov 03 00:25:05.823209 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:05.822218    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f3806a-dockercfg-czmbk\""
Nov 03 00:25:05.885034 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:05.833187    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgzpr\" (UniqueName: \"kubernetes.io/projected/8169d8f5-4823-4d28-ab3e-38fad6ee21bc-kube-api-access-lgzpr\") pod \"payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm\" (UID: \"8169d8f5-4823-4d28-ab3e-38fad6ee21bc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm"
Nov 03 00:25:05.885034 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:05.833422    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8169d8f5-4823-4d28-ab3e-38fad6ee21bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm\" (UID: \"8169d8f5-4823-4d28-ab3e-38fad6ee21bc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm"
Nov 03 00:25:05.939381 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:05.936523    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgzpr\" (UniqueName: \"kubernetes.io/projected/8169d8f5-4823-4d28-ab3e-38fad6ee21bc-kube-api-access-lgzpr\") pod \"payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm\" (UID: \"8169d8f5-4823-4d28-ab3e-38fad6ee21bc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm"
Nov 03 00:25:05.939381 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:05.936736    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8169d8f5-4823-4d28-ab3e-38fad6ee21bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm\" (UID: \"8169d8f5-4823-4d28-ab3e-38fad6ee21bc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm"
Nov 03 00:25:05.944766 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:05.944726    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8169d8f5-4823-4d28-ab3e-38fad6ee21bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm\" (UID: \"8169d8f5-4823-4d28-ab3e-38fad6ee21bc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm"
Nov 03 00:25:05.971595 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:05.971308    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m"]
Nov 03 00:25:05.979806 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:05.979780    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m"
Nov 03 00:25:05.981008 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:05.980574    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m"]
Nov 03 00:25:05.986063 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:05.986042    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgzpr\" (UniqueName: \"kubernetes.io/projected/8169d8f5-4823-4d28-ab3e-38fad6ee21bc-kube-api-access-lgzpr\") pod \"payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm\" (UID: \"8169d8f5-4823-4d28-ab3e-38fad6ee21bc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm"
Nov 03 00:25:06.002642 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:06.002617    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm"
Nov 03 00:25:06.142702 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:06.142668    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxgw8\" (UniqueName: \"kubernetes.io/projected/8569e61f-5ea4-456a-a53a-adad1b4a7ae4-kube-api-access-cxgw8\") pod \"payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m\" (UID: \"8569e61f-5ea4-456a-a53a-adad1b4a7ae4\") " pod="openshift-ingress/payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m"
Nov 03 00:25:06.143156 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:06.143127    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8569e61f-5ea4-456a-a53a-adad1b4a7ae4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m\" (UID: \"8569e61f-5ea4-456a-a53a-adad1b4a7ae4\") " pod="openshift-ingress/payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m"
Nov 03 00:25:06.244738 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:06.244668    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8569e61f-5ea4-456a-a53a-adad1b4a7ae4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m\" (UID: \"8569e61f-5ea4-456a-a53a-adad1b4a7ae4\") " pod="openshift-ingress/payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m"
Nov 03 00:25:06.244862 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:06.244780    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxgw8\" (UniqueName: \"kubernetes.io/projected/8569e61f-5ea4-456a-a53a-adad1b4a7ae4-kube-api-access-cxgw8\") pod \"payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m\" (UID: \"8569e61f-5ea4-456a-a53a-adad1b4a7ae4\") " pod="openshift-ingress/payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m"
Nov 03 00:25:06.245409 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:06.245388    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8569e61f-5ea4-456a-a53a-adad1b4a7ae4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m\" (UID: \"8569e61f-5ea4-456a-a53a-adad1b4a7ae4\") " pod="openshift-ingress/payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m"
Nov 03 00:25:06.262530 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:06.262493    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxgw8\" (UniqueName: \"kubernetes.io/projected/8569e61f-5ea4-456a-a53a-adad1b4a7ae4-kube-api-access-cxgw8\") pod \"payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m\" (UID: \"8569e61f-5ea4-456a-a53a-adad1b4a7ae4\") " pod="openshift-ingress/payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m"
Nov 03 00:25:06.327472 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:25:06.327448    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8169d8f5_4823_4d28_ab3e_38fad6ee21bc.slice/crio-2316d75bba27c78a1965ee382bc802176de1d9828db9f2ea33e9bc1b07f2f0fa WatchSource:0}: Error finding container 2316d75bba27c78a1965ee382bc802176de1d9828db9f2ea33e9bc1b07f2f0fa: Status 404 returned error can't find the container with id 2316d75bba27c78a1965ee382bc802176de1d9828db9f2ea33e9bc1b07f2f0fa
Nov 03 00:25:06.328234 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:06.328181    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m"
Nov 03 00:25:06.328981 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:06.328958    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm"]
Nov 03 00:25:06.641135 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:06.641097    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m"]
Nov 03 00:25:06.641636 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:25:06.641610    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8569e61f_5ea4_456a_a53a_adad1b4a7ae4.slice/crio-3fefe7d35e3c695ee22f5f754ebfcf53cc28b3c2d502b811c1bcf59b9f4e34d9 WatchSource:0}: Error finding container 3fefe7d35e3c695ee22f5f754ebfcf53cc28b3c2d502b811c1bcf59b9f4e34d9: Status 404 returned error can't find the container with id 3fefe7d35e3c695ee22f5f754ebfcf53cc28b3c2d502b811c1bcf59b9f4e34d9
Nov 03 00:25:07.030015 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.029927    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"]
Nov 03 00:25:07.037868 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.037845    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.050371 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.050345    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"]
Nov 03 00:25:07.061947 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.055176    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4aa4e9-openshift-default-dockercfg-fkwlw\""
Nov 03 00:25:07.156457 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.156425    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc956694-af1f-467a-9e91-6a3a16e849ce-istiod-ca-cert\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.156612 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.156569    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-podinfo\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.156718 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.156607    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-token\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.156827 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.156813    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-workload-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.156868 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.156834    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-data\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.156958 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.156933    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scp8m\" (UniqueName: \"kubernetes.io/projected/dc956694-af1f-467a-9e91-6a3a16e849ce-kube-api-access-scp8m\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.157034 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.156999    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-envoy\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.157089 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.157048    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-credential-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.157143 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.157102    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-workload-certs\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.257987 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.257953    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-podinfo\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.258155 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.258005    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-token\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.258155 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.258145    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-workload-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.258293 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.258166    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-data\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.258293 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.258220    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scp8m\" (UniqueName: \"kubernetes.io/projected/dc956694-af1f-467a-9e91-6a3a16e849ce-kube-api-access-scp8m\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.258293 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.258246    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-envoy\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.258293 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.258282    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-credential-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.258543 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.258328    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-workload-certs\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.258543 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.258383    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc956694-af1f-467a-9e91-6a3a16e849ce-istiod-ca-cert\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.258654 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.258574    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-workload-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.258737 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.258710    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-credential-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.258805 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.258751    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-workload-certs\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.258887 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.258870    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-data\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.259060 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.259042    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc956694-af1f-467a-9e91-6a3a16e849ce-istiod-ca-cert\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.260378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.260312    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-envoy\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.260474 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.260441    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-podinfo\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.276039 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.276009    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scp8m\" (UniqueName: \"kubernetes.io/projected/dc956694-af1f-467a-9e91-6a3a16e849ce-kube-api-access-scp8m\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.276039 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.276020    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-token\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.326636 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.326557    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm" event={"ID":"8169d8f5-4823-4d28-ab3e-38fad6ee21bc","Type":"ContainerStarted","Data":"ebf1595383b22bbde529a18d2fce2b540660e627c26fe2195cfcd309d57603ad"}
Nov 03 00:25:07.326636 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.326612    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm" event={"ID":"8169d8f5-4823-4d28-ab3e-38fad6ee21bc","Type":"ContainerStarted","Data":"2316d75bba27c78a1965ee382bc802176de1d9828db9f2ea33e9bc1b07f2f0fa"}
Nov 03 00:25:07.326829 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.326640    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm"
Nov 03 00:25:07.328415 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.328388    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m" event={"ID":"8569e61f-5ea4-456a-a53a-adad1b4a7ae4","Type":"ContainerStarted","Data":"5bccba9351fdc9c0e8527a17c28ffc09b22996c830d17b767c9c332ed4420128"}
Nov 03 00:25:07.328535 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.328437    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m" event={"ID":"8569e61f-5ea4-456a-a53a-adad1b4a7ae4","Type":"ContainerStarted","Data":"3fefe7d35e3c695ee22f5f754ebfcf53cc28b3c2d502b811c1bcf59b9f4e34d9"}
Nov 03 00:25:07.328535 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.328463    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m"
Nov 03 00:25:07.357640 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.357607    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:07.361381 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.361118    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm" podStartSLOduration=2.361104217 podStartE2EDuration="2.361104217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:07.36049728 +0000 UTC m=+1190.240081326" watchObservedRunningTime="2026-11-03 00:25:07.361104217 +0000 UTC m=+1190.240688262"
Nov 03 00:25:07.369389 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:25:07.369353    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:25:07.400197 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.400152    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m" podStartSLOduration=2.400140117 podStartE2EDuration="2.400140117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:07.395680575 +0000 UTC m=+1190.275264618" watchObservedRunningTime="2026-11-03 00:25:07.400140117 +0000 UTC m=+1190.279724161"
Nov 03 00:25:07.694676 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.694639    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"]
Nov 03 00:25:07.705067 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:25:07.705039    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc956694_af1f_467a_9e91_6a3a16e849ce.slice/crio-32c3e0bfa97f3fc5005b9ef54a19c1330d37f0bec040cb98ac2cfdeaba7db91e WatchSource:0}: Error finding container 32c3e0bfa97f3fc5005b9ef54a19c1330d37f0bec040cb98ac2cfdeaba7db91e: Status 404 returned error can't find the container with id 32c3e0bfa97f3fc5005b9ef54a19c1330d37f0bec040cb98ac2cfdeaba7db91e
Nov 03 00:25:07.707549 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.707511    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:25:07.707654 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.707576    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:25:07.707654 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.707603    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:25:07.956491 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.956416    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l"]
Nov 03 00:25:07.986179 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:07.986147    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb"]
Nov 03 00:25:08.360678 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:08.360637    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5" event={"ID":"dc956694-af1f-467a-9e91-6a3a16e849ce","Type":"ContainerStarted","Data":"3b86b2406a429bb58e4af58268ad82dd1a676c41d650a6816a047df8f2756dac"}
Nov 03 00:25:08.361194 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:08.360706    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5" event={"ID":"dc956694-af1f-467a-9e91-6a3a16e849ce","Type":"ContainerStarted","Data":"32c3e0bfa97f3fc5005b9ef54a19c1330d37f0bec040cb98ac2cfdeaba7db91e"}
Nov 03 00:25:08.410969 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:08.410025    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5" podStartSLOduration=2.410003445 podStartE2EDuration="2.410003445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:08.409250646 +0000 UTC m=+1191.288834690" watchObservedRunningTime="2026-11-03 00:25:08.410003445 +0000 UTC m=+1191.289587490"
Nov 03 00:25:09.358143 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:09.358110    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:09.359259 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:09.359230    5012 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 03 00:25:09.359330 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:09.359291    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5" podUID="dc956694-af1f-467a-9e91-6a3a16e849ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 03 00:25:10.358463 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:10.358423    5012 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 03 00:25:10.358852 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:10.358479    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5" podUID="dc956694-af1f-467a-9e91-6a3a16e849ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 03 00:25:11.358234 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:11.358195    5012 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 03 00:25:11.358395 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:11.358258    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5" podUID="dc956694-af1f-467a-9e91-6a3a16e849ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 03 00:25:12.358568 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:12.358531    5012 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 03 00:25:12.359076 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:12.358602    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5" podUID="dc956694-af1f-467a-9e91-6a3a16e849ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 03 00:25:12.956845 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:12.956761    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l" podUID="fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1" containerName="payload-processing" containerID="cri-o://5847cfb1e2820d21607236953a82cb6d439afeab62159a469346d59aa74fcec9" gracePeriod=25
Nov 03 00:25:13.002933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:12.999993    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb" podUID="ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782" containerName="payload-pre-processing" containerID="cri-o://09c4b16907633fa28ff17db1a593b135003a8502b0a4775422c74d2037603577" gracePeriod=25
Nov 03 00:25:13.366962 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:13.360615    5012 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 03 00:25:13.366962 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:13.360687    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5" podUID="dc956694-af1f-467a-9e91-6a3a16e849ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 03 00:25:13.891259 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:13.867702    5012 generic.go:356] "Generic (PLEG): container finished" podID="ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782" containerID="09c4b16907633fa28ff17db1a593b135003a8502b0a4775422c74d2037603577" exitCode=0
Nov 03 00:25:13.891259 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:13.867987    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb" event={"ID":"ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782","Type":"ContainerDied","Data":"09c4b16907633fa28ff17db1a593b135003a8502b0a4775422c74d2037603577"}
Nov 03 00:25:14.148990 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:14.148799    5012 generic.go:356] "Generic (PLEG): container finished" podID="fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1" containerID="5847cfb1e2820d21607236953a82cb6d439afeab62159a469346d59aa74fcec9" exitCode=0
Nov 03 00:25:14.149252 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:14.148922    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l" event={"ID":"fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1","Type":"ContainerDied","Data":"5847cfb1e2820d21607236953a82cb6d439afeab62159a469346d59aa74fcec9"}
Nov 03 00:25:14.363931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:14.363149    5012 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 03 00:25:14.363931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:14.363226    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5" podUID="dc956694-af1f-467a-9e91-6a3a16e849ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 03 00:25:14.563944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:14.547300    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb"
Nov 03 00:25:14.577929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:14.576092    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l"
Nov 03 00:25:14.665545 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:14.665505    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782-plugins-config-volume\") pod \"ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782\" (UID: \"ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782\") "
Nov 03 00:25:14.673127 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:14.673050    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1-plugins-config-volume\") pod \"fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1\" (UID: \"fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1\") "
Nov 03 00:25:14.691932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:14.673395    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782-kube-api-access-c788d\" (UniqueName: \"kubernetes.io/projected/ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782-kube-api-access-c788d\") pod \"ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782\" (UID: \"ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782\") "
Nov 03 00:25:14.692111 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:14.684456    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1-plugins-config-volume" pod "fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1" (UID: "fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:25:14.692242 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:14.692158    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1-kube-api-access-5vns4\" (UniqueName: \"kubernetes.io/projected/fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1-kube-api-access-5vns4\") pod \"fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1\" (UID: \"fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1\") "
Nov 03 00:25:14.717932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:14.693112    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:25:14.732973 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:14.700146    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782-kube-api-access-c788d" pod "ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782" (UID: "ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782"). InnerVolumeSpecName "kube-api-access-c788d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:25:14.733159 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:14.718274    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782-plugins-config-volume" pod "ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782" (UID: "ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:25:14.862933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:14.838265    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:25:14.868291 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:14.868258    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c788d\" (UniqueName: \"kubernetes.io/projected/ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782-kube-api-access-c788d\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:25:14.868523 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:14.868491    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:25:14.871183 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:14.871154    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded"
Nov 03 00:25:14.887119 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:14.887087    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:25:14.994979 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:14.994065    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1-kube-api-access-5vns4" pod "fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1" (UID: "fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1"). InnerVolumeSpecName "kube-api-access-5vns4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:25:15.079772 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:15.079728    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vns4\" (UniqueName: \"kubernetes.io/projected/fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1-kube-api-access-5vns4\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:25:15.083031 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:15.083002    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4da3e6a9-dgsvs"]
Nov 03 00:25:15.105930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:15.099701    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4da3e6a9-dgsvs"]
Nov 03 00:25:15.236151 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:15.234308    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb" event={"ID":"ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782","Type":"ContainerDied","Data":"086ef23eb59353228dec1ee6649d2276326a3e75f4f7aa82214b06d4cb66a80c"}
Nov 03 00:25:15.236151 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:15.234374    5012 scope.go:122] "RemoveContainer" containerID="09c4b16907633fa28ff17db1a593b135003a8502b0a4775422c74d2037603577"
Nov 03 00:25:15.236151 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:15.234558    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb"
Nov 03 00:25:15.304929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:15.302005    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l"
Nov 03 00:25:15.317930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:15.307596    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9173f32c-3ccb-4034-91d4-326e14625e94" path="/var/lib/kubelet/pods/9173f32c-3ccb-4034-91d4-326e14625e94/volumes"
Nov 03 00:25:15.317930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:15.308462    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l" event={"ID":"fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1","Type":"ContainerDied","Data":"2680aed1fcead60f55224428ce7329926f1b3b827c598ab9f905901e0bd1d186"}
Nov 03 00:25:15.401945 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:15.399538    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:15.415939 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:15.403860    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:15.440020 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:15.431979    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:25:15.531609 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:15.531579    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb"]
Nov 03 00:25:15.602077 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:15.599039    5012 scope.go:122] "RemoveContainer" containerID="5847cfb1e2820d21607236953a82cb6d439afeab62159a469346d59aa74fcec9"
Nov 03 00:25:15.614953 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:15.614927    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5c5686-7fb997cfdc-7dllb"]
Nov 03 00:25:15.820088 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:15.820044    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l"]
Nov 03 00:25:15.924986 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:15.921273    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5c5686-77767d966f-pkt5l"]
Nov 03 00:25:16.033007 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:16.032966    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ef73dc5-mlftf"]
Nov 03 00:25:16.038190 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:16.038155    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ef73dc5-mlftf"]
Nov 03 00:25:17.247438 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.247400    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231" path="/var/lib/kubelet/pods/9638d4f5-5ac7-48b0-a2e6-2eed8fa2f231/volumes"
Nov 03 00:25:17.247991 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.247972    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782" path="/var/lib/kubelet/pods/ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782/volumes"
Nov 03 00:25:17.248544 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.248529    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1" path="/var/lib/kubelet/pods/fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1/volumes"
Nov 03 00:25:17.264922 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.264881    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl"]
Nov 03 00:25:17.265559 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.265544    5012 cpu_manager.go:414] "containerMap: removing container" podUID="ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782" containerName="payload-pre-processing"
Nov 03 00:25:17.265610 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.265560    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782" containerName="payload-pre-processing"
Nov 03 00:25:17.265643 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.265619    5012 cpu_manager.go:414] "containerMap: removing container" podUID="fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1" containerName="payload-processing"
Nov 03 00:25:17.265643 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.265625    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1" containerName="payload-processing"
Nov 03 00:25:17.265732 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.265723    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="fccdbbc0-6cb5-4732-8d45-d2c8cfbf9ff1" containerName="payload-processing"
Nov 03 00:25:17.265768 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.265734    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac9a2bf0-8cd4-48ec-bdf5-b1b8417c8782" containerName="payload-pre-processing"
Nov 03 00:25:17.271192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.271174    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl"
Nov 03 00:25:17.291755 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.291733    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl"]
Nov 03 00:25:17.295723 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.295702    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-41e815\""
Nov 03 00:25:17.336255 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.336218    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb"]
Nov 03 00:25:17.341259 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.341237    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb"
Nov 03 00:25:17.353532 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.353504    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb"]
Nov 03 00:25:17.382985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.380425    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-41e815-dockercfg-jqnqm\""
Nov 03 00:25:17.484057 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.478229    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv4bg\" (UniqueName: \"kubernetes.io/projected/60be433d-16ea-414a-afb7-d2e2465472b4-kube-api-access-qv4bg\") pod \"payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl\" (UID: \"60be433d-16ea-414a-afb7-d2e2465472b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl"
Nov 03 00:25:17.484057 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.479528    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e974706-7e0d-4dc1-8ccc-712a52554ef9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb\" (UID: \"8e974706-7e0d-4dc1-8ccc-712a52554ef9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb"
Nov 03 00:25:17.484057 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.479683    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrrrh\" (UniqueName: \"kubernetes.io/projected/8e974706-7e0d-4dc1-8ccc-712a52554ef9-kube-api-access-rrrrh\") pod \"payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb\" (UID: \"8e974706-7e0d-4dc1-8ccc-712a52554ef9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb"
Nov 03 00:25:17.484057 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.479837    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60be433d-16ea-414a-afb7-d2e2465472b4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl\" (UID: \"60be433d-16ea-414a-afb7-d2e2465472b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl"
Nov 03 00:25:17.593768 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.593545    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e974706-7e0d-4dc1-8ccc-712a52554ef9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb\" (UID: \"8e974706-7e0d-4dc1-8ccc-712a52554ef9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb"
Nov 03 00:25:17.593768 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.593606    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrrrh\" (UniqueName: \"kubernetes.io/projected/8e974706-7e0d-4dc1-8ccc-712a52554ef9-kube-api-access-rrrrh\") pod \"payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb\" (UID: \"8e974706-7e0d-4dc1-8ccc-712a52554ef9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb"
Nov 03 00:25:17.593768 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.593670    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60be433d-16ea-414a-afb7-d2e2465472b4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl\" (UID: \"60be433d-16ea-414a-afb7-d2e2465472b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl"
Nov 03 00:25:17.594085 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.593836    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv4bg\" (UniqueName: \"kubernetes.io/projected/60be433d-16ea-414a-afb7-d2e2465472b4-kube-api-access-qv4bg\") pod \"payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl\" (UID: \"60be433d-16ea-414a-afb7-d2e2465472b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl"
Nov 03 00:25:17.601020 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.600987    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e974706-7e0d-4dc1-8ccc-712a52554ef9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb\" (UID: \"8e974706-7e0d-4dc1-8ccc-712a52554ef9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb"
Nov 03 00:25:17.605443 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.605416    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60be433d-16ea-414a-afb7-d2e2465472b4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl\" (UID: \"60be433d-16ea-414a-afb7-d2e2465472b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl"
Nov 03 00:25:17.631606 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.631569    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrrrh\" (UniqueName: \"kubernetes.io/projected/8e974706-7e0d-4dc1-8ccc-712a52554ef9-kube-api-access-rrrrh\") pod \"payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb\" (UID: \"8e974706-7e0d-4dc1-8ccc-712a52554ef9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb"
Nov 03 00:25:17.638122 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.638088    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv4bg\" (UniqueName: \"kubernetes.io/projected/60be433d-16ea-414a-afb7-d2e2465472b4-kube-api-access-qv4bg\") pod \"payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl\" (UID: \"60be433d-16ea-414a-afb7-d2e2465472b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl"
Nov 03 00:25:17.661106 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.661015    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb"
Nov 03 00:25:17.723294 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.723253    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 00:25:17.729374 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.729350    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 00:25:17.729792 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.729769    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:25:17.737431 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.737404    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:25:17.886012 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.885983    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl"
Nov 03 00:25:17.941613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.941585    5012 scope.go:122] "RemoveContainer" containerID="1685539c0d50e74c3743c0bd821a8972ac56291f468197ca852ef6755b98bd75"
Nov 03 00:25:18.000316 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:17.997567    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb"]
Nov 03 00:25:18.000316 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:25:17.998056    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e974706_7e0d_4dc1_8ccc_712a52554ef9.slice/crio-56e73e21450e5130151f3f4009235da59725a7cf1dafa450f66ced41574c8453 WatchSource:0}: Error finding container 56e73e21450e5130151f3f4009235da59725a7cf1dafa450f66ced41574c8453: Status 404 returned error can't find the container with id 56e73e21450e5130151f3f4009235da59725a7cf1dafa450f66ced41574c8453
Nov 03 00:25:18.204054 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:25:18.200502    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60be433d_16ea_414a_afb7_d2e2465472b4.slice/crio-76158754dec1cfe572438fb40c37d99d55816ea04d346b7c1648cb42318285ae WatchSource:0}: Error finding container 76158754dec1cfe572438fb40c37d99d55816ea04d346b7c1648cb42318285ae: Status 404 returned error can't find the container with id 76158754dec1cfe572438fb40c37d99d55816ea04d346b7c1648cb42318285ae
Nov 03 00:25:18.204054 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:18.203824    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl"]
Nov 03 00:25:18.345398 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:18.345363    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb" event={"ID":"8e974706-7e0d-4dc1-8ccc-712a52554ef9","Type":"ContainerStarted","Data":"f79a08ddacf14ed2a188950b285a4366e613ad949216ba6b1eaa2cef8700c523"}
Nov 03 00:25:18.345756 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:18.345435    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb"
Nov 03 00:25:18.345756 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:18.345460    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb" event={"ID":"8e974706-7e0d-4dc1-8ccc-712a52554ef9","Type":"ContainerStarted","Data":"56e73e21450e5130151f3f4009235da59725a7cf1dafa450f66ced41574c8453"}
Nov 03 00:25:18.347010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:18.346986    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl" event={"ID":"60be433d-16ea-414a-afb7-d2e2465472b4","Type":"ContainerStarted","Data":"05bb992b05caffa658ceb4596319ab70c7d86074b916463a410bec4931a1880a"}
Nov 03 00:25:18.347144 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:18.347032    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl" event={"ID":"60be433d-16ea-414a-afb7-d2e2465472b4","Type":"ContainerStarted","Data":"76158754dec1cfe572438fb40c37d99d55816ea04d346b7c1648cb42318285ae"}
Nov 03 00:25:18.347144 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:18.347066    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl"
Nov 03 00:25:18.363620 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:18.363600    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm"
Nov 03 00:25:18.363711 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:18.363670    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m"
Nov 03 00:25:18.382430 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:18.382391    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb" podStartSLOduration=1.3823789 podStartE2EDuration="1.3823789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:18.379367333 +0000 UTC m=+1201.258951378" watchObservedRunningTime="2026-11-03 00:25:18.3823789 +0000 UTC m=+1201.261962965"
Nov 03 00:25:18.414997 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:18.414952    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl" podStartSLOduration=1.4149403999999999 podStartE2EDuration="1.4149404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:18.413403076 +0000 UTC m=+1201.292987119" watchObservedRunningTime="2026-11-03 00:25:18.4149404 +0000 UTC m=+1201.294524443"
Nov 03 00:25:27.099893 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.099856    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"]
Nov 03 00:25:27.112484 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.112455    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.131310 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.131218    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"]
Nov 03 00:25:27.132823 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.132744    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-493ec8-openshift-default-dockercfg-xmmfn\""
Nov 03 00:25:27.186762 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.186727    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-credential-socket\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.186970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.186880    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4accff6-4a81-4c76-9b46-417203ec2921-istiod-ca-cert\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.186970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.186936    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-istio-envoy\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.187101 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.187083    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-workload-certs\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.187158 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.187139    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4accff6-4a81-4c76-9b46-417203ec2921-istio-podinfo\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.187196 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.187177    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4accff6-4a81-4c76-9b46-417203ec2921-istio-token\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.187230 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.187207    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-istio-data\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.187324 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.187308    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5959\" (UniqueName: \"kubernetes.io/projected/b4accff6-4a81-4c76-9b46-417203ec2921-kube-api-access-h5959\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.187426 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.187412    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-workload-socket\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.288849 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.288818    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5959\" (UniqueName: \"kubernetes.io/projected/b4accff6-4a81-4c76-9b46-417203ec2921-kube-api-access-h5959\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.289034 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.288895    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-workload-socket\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.289034 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.288971    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-credential-socket\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.289174 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.289153    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4accff6-4a81-4c76-9b46-417203ec2921-istiod-ca-cert\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.289245 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.289193    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-istio-envoy\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.289305 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.289276    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-credential-socket\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.289305 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.289281    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-workload-certs\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.289412 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.289357    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4accff6-4a81-4c76-9b46-417203ec2921-istio-podinfo\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.289412 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.289362    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-workload-socket\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.289412 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.289399    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4accff6-4a81-4c76-9b46-417203ec2921-istio-token\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.289565 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.289430    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-istio-data\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.289687 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.289647    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-workload-certs\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.289875 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.289850    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-istio-data\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.290008 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.289874    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4accff6-4a81-4c76-9b46-417203ec2921-istiod-ca-cert\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.291600 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.291573    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-istio-envoy\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.291866 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.291847    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4accff6-4a81-4c76-9b46-417203ec2921-istio-podinfo\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.308923 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.308888    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4accff6-4a81-4c76-9b46-417203ec2921-istio-token\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.309323 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.309306    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5959\" (UniqueName: \"kubernetes.io/projected/b4accff6-4a81-4c76-9b46-417203ec2921-kube-api-access-h5959\") pod \"e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") " pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.444987 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.444896    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:27.965079 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:25:27.965051    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4accff6_4a81_4c76_9b46_417203ec2921.slice/crio-b51d190ac0d70eb9392a64886674609f49a0c6c634cac139fce35a085c1ac8d6 WatchSource:0}: Error finding container b51d190ac0d70eb9392a64886674609f49a0c6c634cac139fce35a085c1ac8d6: Status 404 returned error can't find the container with id b51d190ac0d70eb9392a64886674609f49a0c6c634cac139fce35a085c1ac8d6
Nov 03 00:25:27.965585 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.965560    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"]
Nov 03 00:25:27.967969 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.967901    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:25:27.968048 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.968004    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:25:27.968048 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:27.968037    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:25:28.454918 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:28.454872    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4" event={"ID":"b4accff6-4a81-4c76-9b46-417203ec2921","Type":"ContainerStarted","Data":"a24155cc71d930475106728162884b1d7f972a0eea49a6ad023877410eaac831"}
Nov 03 00:25:28.455303 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:28.454936    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4" event={"ID":"b4accff6-4a81-4c76-9b46-417203ec2921","Type":"ContainerStarted","Data":"b51d190ac0d70eb9392a64886674609f49a0c6c634cac139fce35a085c1ac8d6"}
Nov 03 00:25:28.507278 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:28.507221    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4" podStartSLOduration=1.507208099 podStartE2EDuration="1.507208099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:28.502468942 +0000 UTC m=+1211.382052982" watchObservedRunningTime="2026-11-03 00:25:28.507208099 +0000 UTC m=+1211.386792142"
Nov 03 00:25:29.359235 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:29.359201    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl"
Nov 03 00:25:29.359723 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:29.359700    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb"
Nov 03 00:25:29.446064 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:29.446037    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:29.451977 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:29.451951    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:29.466210 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:29.466181    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:29.467316 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:29.467291    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:25:30.041006 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:30.040971    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fe2ca7-fj2rz"]
Nov 03 00:25:30.045876 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:30.045848    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fe2ca7-fj2rz"]
Nov 03 00:25:30.789782 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:30.789740    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"]
Nov 03 00:25:30.791000 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:30.790953    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57" podUID="fc06ba19-a467-4dc0-8d91-b272b7151b20" containerName="istio-proxy" containerID="cri-o://a2dc18cf2d5d516449009f80b5c1d95508169d565890ec252389f23f99af8b75" gracePeriod=30
Nov 03 00:25:31.247732 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:31.247692    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7121c2a-6dcf-4671-855b-07bafc66e27d" path="/var/lib/kubelet/pods/d7121c2a-6dcf-4671-855b-07bafc66e27d/volumes"
Nov 03 00:25:36.335979 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.335953    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:25:36.393582 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.393549    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc06ba19-a467-4dc0-8d91-b272b7151b20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc06ba19-a467-4dc0-8d91-b272b7151b20-istiod-ca-cert\") pod \"fc06ba19-a467-4dc0-8d91-b272b7151b20\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") "
Nov 03 00:25:36.393766 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.393675    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-podinfo\") pod \"fc06ba19-a467-4dc0-8d91-b272b7151b20\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") "
Nov 03 00:25:36.393766 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.393726    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc06ba19-a467-4dc0-8d91-b272b7151b20-kube-api-access-zbcrg\" (UniqueName: \"kubernetes.io/projected/fc06ba19-a467-4dc0-8d91-b272b7151b20-kube-api-access-zbcrg\") pod \"fc06ba19-a467-4dc0-8d91-b272b7151b20\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") "
Nov 03 00:25:36.394019 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.393894    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-credential-socket\") pod \"fc06ba19-a467-4dc0-8d91-b272b7151b20\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") "
Nov 03 00:25:36.394019 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.393957    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc06ba19-a467-4dc0-8d91-b272b7151b20-istiod-ca-cert" pod "fc06ba19-a467-4dc0-8d91-b272b7151b20" (UID: "fc06ba19-a467-4dc0-8d91-b272b7151b20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:25:36.394227 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.394204    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-credential-socket" pod "fc06ba19-a467-4dc0-8d91-b272b7151b20" (UID: "fc06ba19-a467-4dc0-8d91-b272b7151b20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:25:36.394416 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.394390    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-envoy\") pod \"fc06ba19-a467-4dc0-8d91-b272b7151b20\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") "
Nov 03 00:25:36.394555 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.394432    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-workload-socket\") pod \"fc06ba19-a467-4dc0-8d91-b272b7151b20\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") "
Nov 03 00:25:36.394555 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.394520    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-data\") pod \"fc06ba19-a467-4dc0-8d91-b272b7151b20\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") "
Nov 03 00:25:36.394555 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.394548    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-workload-certs\") pod \"fc06ba19-a467-4dc0-8d91-b272b7151b20\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") "
Nov 03 00:25:36.394697 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.394596    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-token\" (UniqueName: \"kubernetes.io/projected/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-token\") pod \"fc06ba19-a467-4dc0-8d91-b272b7151b20\" (UID: \"fc06ba19-a467-4dc0-8d91-b272b7151b20\") "
Nov 03 00:25:36.394852 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.394827    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-data" pod "fc06ba19-a467-4dc0-8d91-b272b7151b20" (UID: "fc06ba19-a467-4dc0-8d91-b272b7151b20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:25:36.395515 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.395070    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-workload-certs" pod "fc06ba19-a467-4dc0-8d91-b272b7151b20" (UID: "fc06ba19-a467-4dc0-8d91-b272b7151b20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:25:36.395515 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.395470    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:25:36.395515 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.395494    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:25:36.395515 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.395509    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc06ba19-a467-4dc0-8d91-b272b7151b20-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:25:36.395818 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.395523    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:25:36.395979 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.395902    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-workload-socket" pod "fc06ba19-a467-4dc0-8d91-b272b7151b20" (UID: "fc06ba19-a467-4dc0-8d91-b272b7151b20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:25:36.397211 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.397183    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-podinfo" pod "fc06ba19-a467-4dc0-8d91-b272b7151b20" (UID: "fc06ba19-a467-4dc0-8d91-b272b7151b20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:25:36.397408 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.397370    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-token" pod "fc06ba19-a467-4dc0-8d91-b272b7151b20" (UID: "fc06ba19-a467-4dc0-8d91-b272b7151b20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:25:36.399124 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.399039    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-envoy" pod "fc06ba19-a467-4dc0-8d91-b272b7151b20" (UID: "fc06ba19-a467-4dc0-8d91-b272b7151b20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:25:36.399124 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.399085    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc06ba19-a467-4dc0-8d91-b272b7151b20-kube-api-access-zbcrg" pod "fc06ba19-a467-4dc0-8d91-b272b7151b20" (UID: "fc06ba19-a467-4dc0-8d91-b272b7151b20"). InnerVolumeSpecName "kube-api-access-zbcrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:25:36.497630 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.497539    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:25:36.497630 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.497583    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbcrg\" (UniqueName: \"kubernetes.io/projected/fc06ba19-a467-4dc0-8d91-b272b7151b20-kube-api-access-zbcrg\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:25:36.497630 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.497595    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:25:36.497630 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.497605    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc06ba19-a467-4dc0-8d91-b272b7151b20-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:25:36.497630 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.497614    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc06ba19-a467-4dc0-8d91-b272b7151b20-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:25:36.534151 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.534115    5012 generic.go:356] "Generic (PLEG): container finished" podID="fc06ba19-a467-4dc0-8d91-b272b7151b20" containerID="a2dc18cf2d5d516449009f80b5c1d95508169d565890ec252389f23f99af8b75" exitCode=0
Nov 03 00:25:36.534313 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.534228    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"
Nov 03 00:25:36.534313 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.534250    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57" event={"ID":"fc06ba19-a467-4dc0-8d91-b272b7151b20","Type":"ContainerDied","Data":"a2dc18cf2d5d516449009f80b5c1d95508169d565890ec252389f23f99af8b75"}
Nov 03 00:25:36.534398 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.534317    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57" event={"ID":"fc06ba19-a467-4dc0-8d91-b272b7151b20","Type":"ContainerDied","Data":"2dc1f23bb30526f91898da21ebe5aa68ec1e9bfe5469ac41c50a2987ff6e0bd9"}
Nov 03 00:25:36.534398 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.534349    5012 scope.go:122] "RemoveContainer" containerID="a2dc18cf2d5d516449009f80b5c1d95508169d565890ec252389f23f99af8b75"
Nov 03 00:25:36.550059 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.550036    5012 scope.go:122] "RemoveContainer" containerID="3937c38e9ae7805dd91d332da72f7a87413f6d7a79b53ba2b36e806ec73176cf"
Nov 03 00:25:36.561383 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.561356    5012 scope.go:122] "RemoveContainer" containerID="a2dc18cf2d5d516449009f80b5c1d95508169d565890ec252389f23f99af8b75"
Nov 03 00:25:36.561658 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:25:36.561637    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2dc18cf2d5d516449009f80b5c1d95508169d565890ec252389f23f99af8b75\": container with ID starting with a2dc18cf2d5d516449009f80b5c1d95508169d565890ec252389f23f99af8b75 not found: ID does not exist" containerID="a2dc18cf2d5d516449009f80b5c1d95508169d565890ec252389f23f99af8b75"
Nov 03 00:25:36.561754 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.561665    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2dc18cf2d5d516449009f80b5c1d95508169d565890ec252389f23f99af8b75"} err="failed to get container status \"a2dc18cf2d5d516449009f80b5c1d95508169d565890ec252389f23f99af8b75\": rpc error: code = NotFound desc = could not find container \"a2dc18cf2d5d516449009f80b5c1d95508169d565890ec252389f23f99af8b75\": container with ID starting with a2dc18cf2d5d516449009f80b5c1d95508169d565890ec252389f23f99af8b75 not found: ID does not exist"
Nov 03 00:25:36.561754 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.561683    5012 scope.go:122] "RemoveContainer" containerID="3937c38e9ae7805dd91d332da72f7a87413f6d7a79b53ba2b36e806ec73176cf"
Nov 03 00:25:36.561939 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:25:36.561902    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3937c38e9ae7805dd91d332da72f7a87413f6d7a79b53ba2b36e806ec73176cf\": container with ID starting with 3937c38e9ae7805dd91d332da72f7a87413f6d7a79b53ba2b36e806ec73176cf not found: ID does not exist" containerID="3937c38e9ae7805dd91d332da72f7a87413f6d7a79b53ba2b36e806ec73176cf"
Nov 03 00:25:36.562040 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.561946    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3937c38e9ae7805dd91d332da72f7a87413f6d7a79b53ba2b36e806ec73176cf"} err="failed to get container status \"3937c38e9ae7805dd91d332da72f7a87413f6d7a79b53ba2b36e806ec73176cf\": rpc error: code = NotFound desc = could not find container \"3937c38e9ae7805dd91d332da72f7a87413f6d7a79b53ba2b36e806ec73176cf\": container with ID starting with 3937c38e9ae7805dd91d332da72f7a87413f6d7a79b53ba2b36e806ec73176cf not found: ID does not exist"
Nov 03 00:25:36.611736 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.611704    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"]
Nov 03 00:25:36.617482 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:36.617456    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5c5686-openshift-default-7d47fd95b7-tvx57"]
Nov 03 00:25:37.245520 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:37.245484    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc06ba19-a467-4dc0-8d91-b272b7151b20" path="/var/lib/kubelet/pods/fc06ba19-a467-4dc0-8d91-b272b7151b20/volumes"
Nov 03 00:25:37.369353 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:25:37.369308    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:25:44.027114 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:44.027081    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c387219-4c629"]
Nov 03 00:25:44.036835 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:44.036804    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c387219-4c629"]
Nov 03 00:25:45.245101 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.245069    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1fa66d2-1746-4308-bc2a-1f909da2322e" path="/var/lib/kubelet/pods/c1fa66d2-1746-4308-bc2a-1f909da2322e/volumes"
Nov 03 00:25:45.572063 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.571983    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"]
Nov 03 00:25:45.573244 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.573213    5012 cpu_manager.go:414] "containerMap: removing container" podUID="fc06ba19-a467-4dc0-8d91-b272b7151b20" containerName="istio-proxy"
Nov 03 00:25:45.573244 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.573239    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc06ba19-a467-4dc0-8d91-b272b7151b20" containerName="istio-proxy"
Nov 03 00:25:45.573452 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.573287    5012 cpu_manager.go:414] "containerMap: removing container" podUID="fc06ba19-a467-4dc0-8d91-b272b7151b20" containerName="istio-proxy"
Nov 03 00:25:45.573452 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.573295    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc06ba19-a467-4dc0-8d91-b272b7151b20" containerName="istio-proxy"
Nov 03 00:25:45.573563 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.573459    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc06ba19-a467-4dc0-8d91-b272b7151b20" containerName="istio-proxy"
Nov 03 00:25:45.580190 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.580164    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.597398 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.597373    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-cb05ae-openshift-default-dockercfg-b4l7c\""
Nov 03 00:25:45.600922 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.600881    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"]
Nov 03 00:25:45.694022 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.693991    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d88041b1-e790-4f59-aee5-a053132ef8a4-istiod-ca-cert\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.694182 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.694123    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-podinfo\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.694265 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.694242    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-workload-certs\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.694326 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.694296    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-token\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.694416 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.694394    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-credential-socket\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.694460 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.694443    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-workload-socket\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.694507 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.694485    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l8jg\" (UniqueName: \"kubernetes.io/projected/d88041b1-e790-4f59-aee5-a053132ef8a4-kube-api-access-2l8jg\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.694573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.694556    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-data\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.694636 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.694617    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-envoy\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.810999 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.810962    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d88041b1-e790-4f59-aee5-a053132ef8a4-istiod-ca-cert\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.833987 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.825306    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-podinfo\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.837133 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.825158    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d88041b1-e790-4f59-aee5-a053132ef8a4-istiod-ca-cert\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.837882 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.837402    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-workload-certs\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.851937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.837798    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-workload-certs\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.853754 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.838067    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-token\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.853977 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.853959    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-credential-socket\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.854171 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.854143    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-workload-socket\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.855014 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.854982    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l8jg\" (UniqueName: \"kubernetes.io/projected/d88041b1-e790-4f59-aee5-a053132ef8a4-kube-api-access-2l8jg\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.855199 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.855179    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-data\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.855353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.855335    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-envoy\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.856349 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.854616    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-workload-socket\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.856508 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.854900    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-credential-socket\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.868533 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.868508    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-podinfo\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.870903 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.869117    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-data\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.885990 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.882615    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-envoy\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.909440 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.907969    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-token\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:45.909440 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:45.908560    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l8jg\" (UniqueName: \"kubernetes.io/projected/d88041b1-e790-4f59-aee5-a053132ef8a4-kube-api-access-2l8jg\") pod \"e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") " pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:46.197572 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:46.197532    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:46.527584 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:25:46.527554    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88041b1_e790_4f59_aee5_a053132ef8a4.slice/crio-a4aa4dbd4b75abf695b322f3576a9bf1ce24042c37de74acc42465b856f0afbb WatchSource:0}: Error finding container a4aa4dbd4b75abf695b322f3576a9bf1ce24042c37de74acc42465b856f0afbb: Status 404 returned error can't find the container with id a4aa4dbd4b75abf695b322f3576a9bf1ce24042c37de74acc42465b856f0afbb
Nov 03 00:25:46.528258 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:46.528187    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"]
Nov 03 00:25:46.530243 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:46.530212    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:25:46.530308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:46.530272    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:25:46.530308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:46.530301    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:25:46.659036 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:46.659000    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps" event={"ID":"d88041b1-e790-4f59-aee5-a053132ef8a4","Type":"ContainerStarted","Data":"50980e2e32c08dfcdcc3371c2c8831bc59936d37d3c65d0d703fa34607169e9c"}
Nov 03 00:25:46.659188 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:46.659066    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps" event={"ID":"d88041b1-e790-4f59-aee5-a053132ef8a4","Type":"ContainerStarted","Data":"a4aa4dbd4b75abf695b322f3576a9bf1ce24042c37de74acc42465b856f0afbb"}
Nov 03 00:25:46.701683 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:46.701634    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps" podStartSLOduration=1.701620096 podStartE2EDuration="1.701620096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:46.69764315 +0000 UTC m=+1229.577227194" watchObservedRunningTime="2026-11-03 00:25:46.701620096 +0000 UTC m=+1229.581204138"
Nov 03 00:25:47.200741 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:47.200706    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:47.206807 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:47.206777    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:47.667576 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:47.667549    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:47.668466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:47.668446    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:25:50.232835 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:50.232794    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"]
Nov 03 00:25:50.233399 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:50.233103    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5" podUID="4a15f2a8-2b21-454f-aa1e-8cd7c2026efa" containerName="istio-proxy" containerID="cri-o://0dc4924e0c292faf03cacbabb55997c51eeeec6f9ac406b65b113833029b14c2" gracePeriod=30
Nov 03 00:25:50.760779 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:50.760743    5012 patch_prober.go:29] interesting pod/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:25:50.760959 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:50.760802    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5" podUID="4a15f2a8-2b21-454f-aa1e-8cd7c2026efa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:25:51.461059 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:51.461027    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9"]
Nov 03 00:25:51.461946 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:51.461904    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc06ba19-a467-4dc0-8d91-b272b7151b20" containerName="istio-proxy"
Nov 03 00:25:51.466996 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:51.466954    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9"
Nov 03 00:25:51.489078 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:51.489057    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cb05ae\""
Nov 03 00:25:51.512958 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:51.512935    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9"]
Nov 03 00:25:51.653116 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:51.653072    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdccc\" (UniqueName: \"kubernetes.io/projected/ebbddfa4-66d5-49ae-81c0-b8f5ae6463de-kube-api-access-bdccc\") pod \"payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9\" (UID: \"ebbddfa4-66d5-49ae-81c0-b8f5ae6463de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9"
Nov 03 00:25:51.653281 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:51.653215    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebbddfa4-66d5-49ae-81c0-b8f5ae6463de-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9\" (UID: \"ebbddfa4-66d5-49ae-81c0-b8f5ae6463de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9"
Nov 03 00:25:51.755024 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:51.754939    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdccc\" (UniqueName: \"kubernetes.io/projected/ebbddfa4-66d5-49ae-81c0-b8f5ae6463de-kube-api-access-bdccc\") pod \"payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9\" (UID: \"ebbddfa4-66d5-49ae-81c0-b8f5ae6463de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9"
Nov 03 00:25:51.755024 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:51.755009    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebbddfa4-66d5-49ae-81c0-b8f5ae6463de-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9\" (UID: \"ebbddfa4-66d5-49ae-81c0-b8f5ae6463de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9"
Nov 03 00:25:51.755778 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:51.755752    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebbddfa4-66d5-49ae-81c0-b8f5ae6463de-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9\" (UID: \"ebbddfa4-66d5-49ae-81c0-b8f5ae6463de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9"
Nov 03 00:25:51.773149 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:51.773123    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdccc\" (UniqueName: \"kubernetes.io/projected/ebbddfa4-66d5-49ae-81c0-b8f5ae6463de-kube-api-access-bdccc\") pod \"payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9\" (UID: \"ebbddfa4-66d5-49ae-81c0-b8f5ae6463de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9"
Nov 03 00:25:51.788631 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:51.788577    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9"
Nov 03 00:25:52.179992 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.179952    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"]
Nov 03 00:25:52.194687 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.192283    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.214211 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.214180    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"]
Nov 03 00:25:52.312667 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.312596    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9"]
Nov 03 00:25:52.312667 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:25:52.312644    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebbddfa4_66d5_49ae_81c0_b8f5ae6463de.slice/crio-4591acfeda437a1316ae42205676757f0012ffb39b09d987c6c2f6e44e2ca005 WatchSource:0}: Error finding container 4591acfeda437a1316ae42205676757f0012ffb39b09d987c6c2f6e44e2ca005: Status 404 returned error can't find the container with id 4591acfeda437a1316ae42205676757f0012ffb39b09d987c6c2f6e44e2ca005
Nov 03 00:25:52.370952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.370888    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-data\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.371057 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.371025    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istiod-ca-cert\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.371108 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.371077    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-podinfo\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.371192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.371180    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-workload-certs\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.371242 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.371225    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-envoy\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.371301 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.371250    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-workload-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.371352 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.371332    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbp2x\" (UniqueName: \"kubernetes.io/projected/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-kube-api-access-qbp2x\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.371468 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.371450    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-token\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.371522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.371508    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-credential-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.472643 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.472604    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-data\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.473309 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.472757    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istiod-ca-cert\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.473309 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.472827    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-podinfo\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.473309 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.472904    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-workload-certs\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.473309 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.472977    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-data\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.473309 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.472991    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-envoy\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.473309 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.473024    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-workload-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.473309 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.473050    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbp2x\" (UniqueName: \"kubernetes.io/projected/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-kube-api-access-qbp2x\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.473309 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.473147    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-token\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.473309 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.473227    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-credential-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.473837 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.473228    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-workload-certs\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.473837 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.473580    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-workload-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.473837 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.473604    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istiod-ca-cert\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.473837 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.473728    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-credential-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.475126 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.475104    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-envoy\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.475301 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.475282    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-podinfo\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.492687 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.492661    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbp2x\" (UniqueName: \"kubernetes.io/projected/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-kube-api-access-qbp2x\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.492784 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.492760    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-token\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.512286 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.512262    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:52.721467 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.721380    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9" event={"ID":"ebbddfa4-66d5-49ae-81c0-b8f5ae6463de","Type":"ContainerStarted","Data":"248b2e281840cea49897390a1375fc0a0722d5f4e822200730ed921cbf350253"}
Nov 03 00:25:52.721467 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.721450    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9" event={"ID":"ebbddfa4-66d5-49ae-81c0-b8f5ae6463de","Type":"ContainerStarted","Data":"4591acfeda437a1316ae42205676757f0012ffb39b09d987c6c2f6e44e2ca005"}
Nov 03 00:25:52.721740 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.721479    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9"
Nov 03 00:25:52.759474 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.759414    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9" podStartSLOduration=1.7594002789999998 podStartE2EDuration="1.759400279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:52.756769262 +0000 UTC m=+1235.636353329" watchObservedRunningTime="2026-11-03 00:25:52.759400279 +0000 UTC m=+1235.638984352"
Nov 03 00:25:52.836127 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:25:52.836096    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f0a50c_b1c0_4fe7_aa15_b3050d4829f7.slice/crio-4cfc8918f32abb505f9cba0d5a4bbbae1b9fb0da8dedf74c23c97eec9ca178a3 WatchSource:0}: Error finding container 4cfc8918f32abb505f9cba0d5a4bbbae1b9fb0da8dedf74c23c97eec9ca178a3: Status 404 returned error can't find the container with id 4cfc8918f32abb505f9cba0d5a4bbbae1b9fb0da8dedf74c23c97eec9ca178a3
Nov 03 00:25:52.838455 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.838422    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"]
Nov 03 00:25:52.839132 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.839096    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:25:52.839192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.839177    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:25:52.839231 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:52.839218    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:25:53.732261 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:53.732222    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6" event={"ID":"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7","Type":"ContainerStarted","Data":"f97a9eb5abe21f5fbc65159d71ff6cf4e3c2a373adbac095cdeadfb52288754c"}
Nov 03 00:25:53.732697 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:53.732279    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6" event={"ID":"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7","Type":"ContainerStarted","Data":"4cfc8918f32abb505f9cba0d5a4bbbae1b9fb0da8dedf74c23c97eec9ca178a3"}
Nov 03 00:25:53.785444 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:53.785376    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6" podStartSLOduration=1.785359357 podStartE2EDuration="1.785359357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:53.780455864 +0000 UTC m=+1236.660039908" watchObservedRunningTime="2026-11-03 00:25:53.785359357 +0000 UTC m=+1236.664943403"
Nov 03 00:25:54.512667 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:54.512627    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:25:54.514069 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:54.514032    5012 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 03 00:25:54.514192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:54.514093    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6" podUID="88f0a50c-b1c0-4fe7-aa15-b3050d4829f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 03 00:25:55.028186 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.028149    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e803ae6-h8t8f"]
Nov 03 00:25:55.037811 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.037784    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e803ae6-h8t8f"]
Nov 03 00:25:55.245792 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.245757    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88595eae-3638-43d7-83b9-77a1253b2879" path="/var/lib/kubelet/pods/88595eae-3638-43d7-83b9-77a1253b2879/volumes"
Nov 03 00:25:55.521696 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.515567    5012 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 03 00:25:55.521696 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.515620    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6" podUID="88f0a50c-b1c0-4fe7-aa15-b3050d4829f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 03 00:25:55.584568 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.584532    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:25:55.638173 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.635613    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-credential-socket\") pod \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") "
Nov 03 00:25:55.638173 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.635963    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-credential-socket" pod "4a15f2a8-2b21-454f-aa1e-8cd7c2026efa" (UID: "4a15f2a8-2b21-454f-aa1e-8cd7c2026efa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:25:55.638173 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.636100    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-workload-certs\") pod \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") "
Nov 03 00:25:55.638173 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.636370    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-workload-certs" pod "4a15f2a8-2b21-454f-aa1e-8cd7c2026efa" (UID: "4a15f2a8-2b21-454f-aa1e-8cd7c2026efa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:25:55.638173 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.636462    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-kube-api-access-qlbvb\" (UniqueName: \"kubernetes.io/projected/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-kube-api-access-qlbvb\") pod \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") "
Nov 03 00:25:55.638173 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.636867    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-podinfo\") pod \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") "
Nov 03 00:25:55.638173 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.636953    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-workload-socket\") pod \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") "
Nov 03 00:25:55.638173 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.636984    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-token\" (UniqueName: \"kubernetes.io/projected/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-token\") pod \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") "
Nov 03 00:25:55.638173 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.637071    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-data\") pod \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") "
Nov 03 00:25:55.640742 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.638543    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istiod-ca-cert\") pod \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") "
Nov 03 00:25:55.640742 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.638619    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-envoy\") pod \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\" (UID: \"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa\") "
Nov 03 00:25:55.640742 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.639523    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:25:55.640742 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.639546    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:25:55.640742 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.637864    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-data" pod "4a15f2a8-2b21-454f-aa1e-8cd7c2026efa" (UID: "4a15f2a8-2b21-454f-aa1e-8cd7c2026efa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:25:55.640742 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.638259    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-workload-socket" pod "4a15f2a8-2b21-454f-aa1e-8cd7c2026efa" (UID: "4a15f2a8-2b21-454f-aa1e-8cd7c2026efa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:25:55.640742 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.640134    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istiod-ca-cert" pod "4a15f2a8-2b21-454f-aa1e-8cd7c2026efa" (UID: "4a15f2a8-2b21-454f-aa1e-8cd7c2026efa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:25:55.646791 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.646755    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-kube-api-access-qlbvb" pod "4a15f2a8-2b21-454f-aa1e-8cd7c2026efa" (UID: "4a15f2a8-2b21-454f-aa1e-8cd7c2026efa"). InnerVolumeSpecName "kube-api-access-qlbvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:25:55.651273 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.649565    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-token" pod "4a15f2a8-2b21-454f-aa1e-8cd7c2026efa" (UID: "4a15f2a8-2b21-454f-aa1e-8cd7c2026efa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:25:55.651639 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.651460    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-podinfo" pod "4a15f2a8-2b21-454f-aa1e-8cd7c2026efa" (UID: "4a15f2a8-2b21-454f-aa1e-8cd7c2026efa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:25:55.653490 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.652799    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-envoy" pod "4a15f2a8-2b21-454f-aa1e-8cd7c2026efa" (UID: "4a15f2a8-2b21-454f-aa1e-8cd7c2026efa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:25:55.741380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.741351    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlbvb\" (UniqueName: \"kubernetes.io/projected/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-kube-api-access-qlbvb\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:25:55.741380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.741378    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:25:55.741380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.741388    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:25:55.741604 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.741399    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:25:55.741604 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.741407    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:25:55.741604 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.741414    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:25:55.741604 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.741422    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:25:55.766248 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.766216    5012 generic.go:356] "Generic (PLEG): container finished" podID="4a15f2a8-2b21-454f-aa1e-8cd7c2026efa" containerID="0dc4924e0c292faf03cacbabb55997c51eeeec6f9ac406b65b113833029b14c2" exitCode=0
Nov 03 00:25:55.766411 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.766266    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"
Nov 03 00:25:55.766411 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.766298    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5" event={"ID":"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa","Type":"ContainerDied","Data":"0dc4924e0c292faf03cacbabb55997c51eeeec6f9ac406b65b113833029b14c2"}
Nov 03 00:25:55.766411 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.766358    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5" event={"ID":"4a15f2a8-2b21-454f-aa1e-8cd7c2026efa","Type":"ContainerDied","Data":"ca7ff1fc428246267addc730e6292da81868d157de3b1757d7352554d1343876"}
Nov 03 00:25:55.766411 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.766389    5012 scope.go:122] "RemoveContainer" containerID="0dc4924e0c292faf03cacbabb55997c51eeeec6f9ac406b65b113833029b14c2"
Nov 03 00:25:55.779280 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.779258    5012 scope.go:122] "RemoveContainer" containerID="6bab4d0ce3444e7ed0526108ea75429cf427b9fff7c744793d938bf1bd2ac85c"
Nov 03 00:25:55.808635 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.808540    5012 scope.go:122] "RemoveContainer" containerID="0dc4924e0c292faf03cacbabb55997c51eeeec6f9ac406b65b113833029b14c2"
Nov 03 00:25:55.809190 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:25:55.809039    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dc4924e0c292faf03cacbabb55997c51eeeec6f9ac406b65b113833029b14c2\": container with ID starting with 0dc4924e0c292faf03cacbabb55997c51eeeec6f9ac406b65b113833029b14c2 not found: ID does not exist" containerID="0dc4924e0c292faf03cacbabb55997c51eeeec6f9ac406b65b113833029b14c2"
Nov 03 00:25:55.809190 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.809096    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc4924e0c292faf03cacbabb55997c51eeeec6f9ac406b65b113833029b14c2"} err="failed to get container status \"0dc4924e0c292faf03cacbabb55997c51eeeec6f9ac406b65b113833029b14c2\": rpc error: code = NotFound desc = could not find container \"0dc4924e0c292faf03cacbabb55997c51eeeec6f9ac406b65b113833029b14c2\": container with ID starting with 0dc4924e0c292faf03cacbabb55997c51eeeec6f9ac406b65b113833029b14c2 not found: ID does not exist"
Nov 03 00:25:55.809190 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.809121    5012 scope.go:122] "RemoveContainer" containerID="6bab4d0ce3444e7ed0526108ea75429cf427b9fff7c744793d938bf1bd2ac85c"
Nov 03 00:25:55.809736 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:25:55.809673    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bab4d0ce3444e7ed0526108ea75429cf427b9fff7c744793d938bf1bd2ac85c\": container with ID starting with 6bab4d0ce3444e7ed0526108ea75429cf427b9fff7c744793d938bf1bd2ac85c not found: ID does not exist" containerID="6bab4d0ce3444e7ed0526108ea75429cf427b9fff7c744793d938bf1bd2ac85c"
Nov 03 00:25:55.809736 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.809705    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bab4d0ce3444e7ed0526108ea75429cf427b9fff7c744793d938bf1bd2ac85c"} err="failed to get container status \"6bab4d0ce3444e7ed0526108ea75429cf427b9fff7c744793d938bf1bd2ac85c\": rpc error: code = NotFound desc = could not find container \"6bab4d0ce3444e7ed0526108ea75429cf427b9fff7c744793d938bf1bd2ac85c\": container with ID starting with 6bab4d0ce3444e7ed0526108ea75429cf427b9fff7c744793d938bf1bd2ac85c not found: ID does not exist"
Nov 03 00:25:55.843131 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.843101    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"]
Nov 03 00:25:55.850430 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:55.850384    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9f76b708-openshift-default-7664fdbcd9-p8ms5"]
Nov 03 00:25:56.512716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:56.512677    5012 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 03 00:25:56.513219 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:56.512736    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6" podUID="88f0a50c-b1c0-4fe7-aa15-b3050d4829f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 03 00:25:57.250226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:57.250190    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a15f2a8-2b21-454f-aa1e-8cd7c2026efa" path="/var/lib/kubelet/pods/4a15f2a8-2b21-454f-aa1e-8cd7c2026efa/volumes"
Nov 03 00:25:57.526051 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:57.515278    5012 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 03 00:25:57.526051 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:57.515340    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6" podUID="88f0a50c-b1c0-4fe7-aa15-b3050d4829f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 03 00:25:58.076258 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.076222    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655"]
Nov 03 00:25:58.078472 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.078418    5012 cpu_manager.go:414] "containerMap: removing container" podUID="4a15f2a8-2b21-454f-aa1e-8cd7c2026efa" containerName="istio-proxy"
Nov 03 00:25:58.078662 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.078645    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a15f2a8-2b21-454f-aa1e-8cd7c2026efa" containerName="istio-proxy"
Nov 03 00:25:58.079059 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.079027    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a15f2a8-2b21-454f-aa1e-8cd7c2026efa" containerName="istio-proxy"
Nov 03 00:25:58.085539 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.085517    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655"
Nov 03 00:25:58.104015 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.103977    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6"]
Nov 03 00:25:58.106323 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.106292    5012 cpu_manager.go:414] "containerMap: removing container" podUID="4a15f2a8-2b21-454f-aa1e-8cd7c2026efa" containerName="istio-proxy"
Nov 03 00:25:58.106527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.106510    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a15f2a8-2b21-454f-aa1e-8cd7c2026efa" containerName="istio-proxy"
Nov 03 00:25:58.106819 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.106801    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a15f2a8-2b21-454f-aa1e-8cd7c2026efa" containerName="istio-proxy"
Nov 03 00:25:58.112580 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.112561    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6"
Nov 03 00:25:58.122939 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.113469    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-27490b-dockercfg-478z5\""
Nov 03 00:25:58.122939 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.113555    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-27490b\""
Nov 03 00:25:58.124846 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.124817    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d88538c1-dccb-47da-92e9-94738e7be751-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655\" (UID: \"d88538c1-dccb-47da-92e9-94738e7be751\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655"
Nov 03 00:25:58.130419 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.130388    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7mc2\" (UniqueName: \"kubernetes.io/projected/d88538c1-dccb-47da-92e9-94738e7be751-kube-api-access-f7mc2\") pod \"payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655\" (UID: \"d88538c1-dccb-47da-92e9-94738e7be751\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655"
Nov 03 00:25:58.277958 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.239021    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7mc2\" (UniqueName: \"kubernetes.io/projected/d88538c1-dccb-47da-92e9-94738e7be751-kube-api-access-f7mc2\") pod \"payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655\" (UID: \"d88538c1-dccb-47da-92e9-94738e7be751\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655"
Nov 03 00:25:58.277958 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.239193    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d88538c1-dccb-47da-92e9-94738e7be751-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655\" (UID: \"d88538c1-dccb-47da-92e9-94738e7be751\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655"
Nov 03 00:25:58.277958 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.239260    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsd2b\" (UniqueName: \"kubernetes.io/projected/e4cbe05e-6056-425e-b7c8-779e9ab9a6b2-kube-api-access-vsd2b\") pod \"payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6\" (UID: \"e4cbe05e-6056-425e-b7c8-779e9ab9a6b2\") " pod="openshift-ingress/payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6"
Nov 03 00:25:58.277958 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.239309    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4cbe05e-6056-425e-b7c8-779e9ab9a6b2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6\" (UID: \"e4cbe05e-6056-425e-b7c8-779e9ab9a6b2\") " pod="openshift-ingress/payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6"
Nov 03 00:25:58.277958 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.240684    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d88538c1-dccb-47da-92e9-94738e7be751-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655\" (UID: \"d88538c1-dccb-47da-92e9-94738e7be751\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655"
Nov 03 00:25:58.350676 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.350229    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsd2b\" (UniqueName: \"kubernetes.io/projected/e4cbe05e-6056-425e-b7c8-779e9ab9a6b2-kube-api-access-vsd2b\") pod \"payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6\" (UID: \"e4cbe05e-6056-425e-b7c8-779e9ab9a6b2\") " pod="openshift-ingress/payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6"
Nov 03 00:25:58.350676 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.350296    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4cbe05e-6056-425e-b7c8-779e9ab9a6b2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6\" (UID: \"e4cbe05e-6056-425e-b7c8-779e9ab9a6b2\") " pod="openshift-ingress/payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6"
Nov 03 00:25:58.351480 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.351406    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4cbe05e-6056-425e-b7c8-779e9ab9a6b2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6\" (UID: \"e4cbe05e-6056-425e-b7c8-779e9ab9a6b2\") " pod="openshift-ingress/payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6"
Nov 03 00:25:58.379939 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.362145    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7mc2\" (UniqueName: \"kubernetes.io/projected/d88538c1-dccb-47da-92e9-94738e7be751-kube-api-access-f7mc2\") pod \"payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655\" (UID: \"d88538c1-dccb-47da-92e9-94738e7be751\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655"
Nov 03 00:25:58.385154 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.385124    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655"]
Nov 03 00:25:58.393684 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.393632    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6"]
Nov 03 00:25:58.464342 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.461270    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsd2b\" (UniqueName: \"kubernetes.io/projected/e4cbe05e-6056-425e-b7c8-779e9ab9a6b2-kube-api-access-vsd2b\") pod \"payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6\" (UID: \"e4cbe05e-6056-425e-b7c8-779e9ab9a6b2\") " pod="openshift-ingress/payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6"
Nov 03 00:25:58.532180 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.532146    5012 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 03 00:25:58.532737 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.532706    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6" podUID="88f0a50c-b1c0-4fe7-aa15-b3050d4829f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 03 00:25:58.609985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.590247    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655"
Nov 03 00:25:58.689932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:58.686233    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6"
Nov 03 00:25:59.021524 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:25:59.021492    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88538c1_dccb_47da_92e9_94738e7be751.slice/crio-0e16d51d625a20ee2c8faafcc3e9e8a860adf202cd718bcda795c8d935c6957d WatchSource:0}: Error finding container 0e16d51d625a20ee2c8faafcc3e9e8a860adf202cd718bcda795c8d935c6957d: Status 404 returned error can't find the container with id 0e16d51d625a20ee2c8faafcc3e9e8a860adf202cd718bcda795c8d935c6957d
Nov 03 00:25:59.022836 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:59.022807    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655"]
Nov 03 00:25:59.076185 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:25:59.076157    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4cbe05e_6056_425e_b7c8_779e9ab9a6b2.slice/crio-7d42d59c09d4315ec5715f19907f30734616ede2859adb64b719b44bbdc0b7e0 WatchSource:0}: Error finding container 7d42d59c09d4315ec5715f19907f30734616ede2859adb64b719b44bbdc0b7e0: Status 404 returned error can't find the container with id 7d42d59c09d4315ec5715f19907f30734616ede2859adb64b719b44bbdc0b7e0
Nov 03 00:25:59.079476 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:59.079423    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6"]
Nov 03 00:25:59.512991 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:59.512949    5012 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 03 00:25:59.513168 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:59.513023    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6" podUID="88f0a50c-b1c0-4fe7-aa15-b3050d4829f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 03 00:25:59.849244 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:59.849210    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655" event={"ID":"d88538c1-dccb-47da-92e9-94738e7be751","Type":"ContainerStarted","Data":"f9af6154cb885440dee5ea9417aeb0fad01f2b8fa41f867b0dcd170ae9c2838f"}
Nov 03 00:25:59.849773 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:59.849278    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655" event={"ID":"d88538c1-dccb-47da-92e9-94738e7be751","Type":"ContainerStarted","Data":"0e16d51d625a20ee2c8faafcc3e9e8a860adf202cd718bcda795c8d935c6957d"}
Nov 03 00:25:59.849773 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:59.849407    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655"
Nov 03 00:25:59.851287 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:59.851170    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6" event={"ID":"e4cbe05e-6056-425e-b7c8-779e9ab9a6b2","Type":"ContainerStarted","Data":"fb187e9637d8b0d7acd770ad24d9a7928bdd2116044a1706dc485ff37c992ce0"}
Nov 03 00:25:59.851287 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:59.851209    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6" event={"ID":"e4cbe05e-6056-425e-b7c8-779e9ab9a6b2","Type":"ContainerStarted","Data":"7d42d59c09d4315ec5715f19907f30734616ede2859adb64b719b44bbdc0b7e0"}
Nov 03 00:25:59.851287 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:59.851250    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6"
Nov 03 00:25:59.887385 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:59.887326    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655" podStartSLOduration=1.8873065119999999 podStartE2EDuration="1.887306512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:59.883255689 +0000 UTC m=+1242.762839735" watchObservedRunningTime="2026-11-03 00:25:59.887306512 +0000 UTC m=+1242.766890558"
Nov 03 00:25:59.917784 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:25:59.917735    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6" podStartSLOduration=1.917721107 podStartE2EDuration="1.917721107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:59.916688453 +0000 UTC m=+1242.796272496" watchObservedRunningTime="2026-11-03 00:25:59.917721107 +0000 UTC m=+1242.797305184"
Nov 03 00:26:00.517860 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:00.517829    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:26:00.518328 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:00.518309    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:26:00.519040 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:00.519023    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:26:00.686636 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:00.686601    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl"]
Nov 03 00:26:00.726098 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:00.726066    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb"]
Nov 03 00:26:01.336162 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:01.336129    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"]
Nov 03 00:26:01.336955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:01.336882    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5" podUID="dc956694-af1f-467a-9e91-6a3a16e849ce" containerName="istio-proxy" containerID="cri-o://3b86b2406a429bb58e4af58268ad82dd1a676c41d650a6816a047df8f2756dac" gracePeriod=30
Nov 03 00:26:01.338114 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:01.338089    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"]
Nov 03 00:26:01.879089 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:01.879015    5012 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6" secret="" err="secret \"e2e-api-b-4aa4e9-openshift-default-dockercfg-fkwlw\" not found"
Nov 03 00:26:02.888652 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:02.888611    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6" podUID="88f0a50c-b1c0-4fe7-aa15-b3050d4829f7" containerName="istio-proxy" containerID="cri-o://f97a9eb5abe21f5fbc65159d71ff6cf4e3c2a373adbac095cdeadfb52288754c" gracePeriod=30
Nov 03 00:26:03.735604 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:03.735577    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9"
Nov 03 00:26:05.687014 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:05.686972    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl" podUID="60be433d-16ea-414a-afb7-d2e2465472b4" containerName="payload-processing" containerID="cri-o://05bb992b05caffa658ceb4596319ab70c7d86074b916463a410bec4931a1880a" gracePeriod=25
Nov 03 00:26:05.726490 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:05.726429    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb" podUID="8e974706-7e0d-4dc1-8ccc-712a52554ef9" containerName="payload-pre-processing" containerID="cri-o://f79a08ddacf14ed2a188950b285a4366e613ad949216ba6b1eaa2cef8700c523" gracePeriod=25
Nov 03 00:26:05.920934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:05.920113    5012 generic.go:356] "Generic (PLEG): container finished" podID="60be433d-16ea-414a-afb7-d2e2465472b4" containerID="05bb992b05caffa658ceb4596319ab70c7d86074b916463a410bec4931a1880a" exitCode=0
Nov 03 00:26:05.920934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:05.920408    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl" event={"ID":"60be433d-16ea-414a-afb7-d2e2465472b4","Type":"ContainerDied","Data":"05bb992b05caffa658ceb4596319ab70c7d86074b916463a410bec4931a1880a"}
Nov 03 00:26:05.924611 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:05.922628    5012 generic.go:356] "Generic (PLEG): container finished" podID="8e974706-7e0d-4dc1-8ccc-712a52554ef9" containerID="f79a08ddacf14ed2a188950b285a4366e613ad949216ba6b1eaa2cef8700c523" exitCode=0
Nov 03 00:26:05.924611 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:05.922716    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb" event={"ID":"8e974706-7e0d-4dc1-8ccc-712a52554ef9","Type":"ContainerDied","Data":"f79a08ddacf14ed2a188950b285a4366e613ad949216ba6b1eaa2cef8700c523"}
Nov 03 00:26:06.028719 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.028692    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl"
Nov 03 00:26:06.109411 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.109376    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb"
Nov 03 00:26:06.190099 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.190060    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60be433d-16ea-414a-afb7-d2e2465472b4-kube-api-access-qv4bg\" (UniqueName: \"kubernetes.io/projected/60be433d-16ea-414a-afb7-d2e2465472b4-kube-api-access-qv4bg\") pod \"60be433d-16ea-414a-afb7-d2e2465472b4\" (UID: \"60be433d-16ea-414a-afb7-d2e2465472b4\") "
Nov 03 00:26:06.190361 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.190197    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60be433d-16ea-414a-afb7-d2e2465472b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60be433d-16ea-414a-afb7-d2e2465472b4-plugins-config-volume\") pod \"60be433d-16ea-414a-afb7-d2e2465472b4\" (UID: \"60be433d-16ea-414a-afb7-d2e2465472b4\") "
Nov 03 00:26:06.192527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.192493    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60be433d-16ea-414a-afb7-d2e2465472b4-plugins-config-volume" pod "60be433d-16ea-414a-afb7-d2e2465472b4" (UID: "60be433d-16ea-414a-afb7-d2e2465472b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:26:06.193543 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.193517    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60be433d-16ea-414a-afb7-d2e2465472b4-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:06.194407 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.194384    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60be433d-16ea-414a-afb7-d2e2465472b4-kube-api-access-qv4bg" pod "60be433d-16ea-414a-afb7-d2e2465472b4" (UID: "60be433d-16ea-414a-afb7-d2e2465472b4"). InnerVolumeSpecName "kube-api-access-qv4bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:26:06.294997 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.294902    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e974706-7e0d-4dc1-8ccc-712a52554ef9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e974706-7e0d-4dc1-8ccc-712a52554ef9-plugins-config-volume\") pod \"8e974706-7e0d-4dc1-8ccc-712a52554ef9\" (UID: \"8e974706-7e0d-4dc1-8ccc-712a52554ef9\") "
Nov 03 00:26:06.295218 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.295197    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e974706-7e0d-4dc1-8ccc-712a52554ef9-kube-api-access-rrrrh\" (UniqueName: \"kubernetes.io/projected/8e974706-7e0d-4dc1-8ccc-712a52554ef9-kube-api-access-rrrrh\") pod \"8e974706-7e0d-4dc1-8ccc-712a52554ef9\" (UID: \"8e974706-7e0d-4dc1-8ccc-712a52554ef9\") "
Nov 03 00:26:06.295315 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.295289    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e974706-7e0d-4dc1-8ccc-712a52554ef9-plugins-config-volume" pod "8e974706-7e0d-4dc1-8ccc-712a52554ef9" (UID: "8e974706-7e0d-4dc1-8ccc-712a52554ef9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:26:06.295627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.295613    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv4bg\" (UniqueName: \"kubernetes.io/projected/60be433d-16ea-414a-afb7-d2e2465472b4-kube-api-access-qv4bg\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:06.295664 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.295630    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e974706-7e0d-4dc1-8ccc-712a52554ef9-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:06.297180 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.297162    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e974706-7e0d-4dc1-8ccc-712a52554ef9-kube-api-access-rrrrh" pod "8e974706-7e0d-4dc1-8ccc-712a52554ef9" (UID: "8e974706-7e0d-4dc1-8ccc-712a52554ef9"). InnerVolumeSpecName "kube-api-access-rrrrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:26:06.396980 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.396949    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrrrh\" (UniqueName: \"kubernetes.io/projected/8e974706-7e0d-4dc1-8ccc-712a52554ef9-kube-api-access-rrrrh\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:06.616986 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.616962    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:26:06.803492 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.803466    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc956694-af1f-467a-9e91-6a3a16e849ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc956694-af1f-467a-9e91-6a3a16e849ce-istiod-ca-cert\") pod \"dc956694-af1f-467a-9e91-6a3a16e849ce\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") "
Nov 03 00:26:06.803864 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.803523    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-workload-certs\") pod \"dc956694-af1f-467a-9e91-6a3a16e849ce\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") "
Nov 03 00:26:06.803864 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.803575    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-credential-socket\") pod \"dc956694-af1f-467a-9e91-6a3a16e849ce\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") "
Nov 03 00:26:06.803864 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.803620    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-podinfo\") pod \"dc956694-af1f-467a-9e91-6a3a16e849ce\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") "
Nov 03 00:26:06.803864 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.803680    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-envoy\") pod \"dc956694-af1f-467a-9e91-6a3a16e849ce\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") "
Nov 03 00:26:06.803864 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.803742    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-token\" (UniqueName: \"kubernetes.io/projected/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-token\") pod \"dc956694-af1f-467a-9e91-6a3a16e849ce\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") "
Nov 03 00:26:06.803864 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.803848    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc956694-af1f-467a-9e91-6a3a16e849ce-kube-api-access-scp8m\" (UniqueName: \"kubernetes.io/projected/dc956694-af1f-467a-9e91-6a3a16e849ce-kube-api-access-scp8m\") pod \"dc956694-af1f-467a-9e91-6a3a16e849ce\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") "
Nov 03 00:26:06.804310 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.803876    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-workload-certs" pod "dc956694-af1f-467a-9e91-6a3a16e849ce" (UID: "dc956694-af1f-467a-9e91-6a3a16e849ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:06.804310 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.803996    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-credential-socket" pod "dc956694-af1f-467a-9e91-6a3a16e849ce" (UID: "dc956694-af1f-467a-9e91-6a3a16e849ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:06.804310 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.803900    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc956694-af1f-467a-9e91-6a3a16e849ce-istiod-ca-cert" pod "dc956694-af1f-467a-9e91-6a3a16e849ce" (UID: "dc956694-af1f-467a-9e91-6a3a16e849ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:26:06.804310 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.804052    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-workload-socket\") pod \"dc956694-af1f-467a-9e91-6a3a16e849ce\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") "
Nov 03 00:26:06.804310 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.804094    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-data\") pod \"dc956694-af1f-467a-9e91-6a3a16e849ce\" (UID: \"dc956694-af1f-467a-9e91-6a3a16e849ce\") "
Nov 03 00:26:06.804310 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.804247    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-workload-socket" pod "dc956694-af1f-467a-9e91-6a3a16e849ce" (UID: "dc956694-af1f-467a-9e91-6a3a16e849ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:06.804637 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.804416    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-data" pod "dc956694-af1f-467a-9e91-6a3a16e849ce" (UID: "dc956694-af1f-467a-9e91-6a3a16e849ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:06.804755 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.804732    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:06.804824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.804761    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:06.804824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.804777    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc956694-af1f-467a-9e91-6a3a16e849ce-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:06.804824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.804790    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:06.804824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.804803    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:06.806140 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.806114    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-envoy" pod "dc956694-af1f-467a-9e91-6a3a16e849ce" (UID: "dc956694-af1f-467a-9e91-6a3a16e849ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:06.806140 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.806130    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-token" pod "dc956694-af1f-467a-9e91-6a3a16e849ce" (UID: "dc956694-af1f-467a-9e91-6a3a16e849ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:26:06.806292 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.806153    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc956694-af1f-467a-9e91-6a3a16e849ce-kube-api-access-scp8m" pod "dc956694-af1f-467a-9e91-6a3a16e849ce" (UID: "dc956694-af1f-467a-9e91-6a3a16e849ce"). InnerVolumeSpecName "kube-api-access-scp8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:26:06.806442 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.806415    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-podinfo" pod "dc956694-af1f-467a-9e91-6a3a16e849ce" (UID: "dc956694-af1f-467a-9e91-6a3a16e849ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:26:06.906191 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.906112    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scp8m\" (UniqueName: \"kubernetes.io/projected/dc956694-af1f-467a-9e91-6a3a16e849ce-kube-api-access-scp8m\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:06.906191 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.906149    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:06.906191 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.906165    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:06.906191 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.906178    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc956694-af1f-467a-9e91-6a3a16e849ce-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:06.933089 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.933060    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb"
Nov 03 00:26:06.933089 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.933085    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb" event={"ID":"8e974706-7e0d-4dc1-8ccc-712a52554ef9","Type":"ContainerDied","Data":"56e73e21450e5130151f3f4009235da59725a7cf1dafa450f66ced41574c8453"}
Nov 03 00:26:06.933317 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.933165    5012 scope.go:122] "RemoveContainer" containerID="f79a08ddacf14ed2a188950b285a4366e613ad949216ba6b1eaa2cef8700c523"
Nov 03 00:26:06.934760 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.934729    5012 generic.go:356] "Generic (PLEG): container finished" podID="dc956694-af1f-467a-9e91-6a3a16e849ce" containerID="3b86b2406a429bb58e4af58268ad82dd1a676c41d650a6816a047df8f2756dac" exitCode=0
Nov 03 00:26:06.934878 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.934769    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5" event={"ID":"dc956694-af1f-467a-9e91-6a3a16e849ce","Type":"ContainerDied","Data":"3b86b2406a429bb58e4af58268ad82dd1a676c41d650a6816a047df8f2756dac"}
Nov 03 00:26:06.934878 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.934797    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"
Nov 03 00:26:06.934878 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.934827    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5" event={"ID":"dc956694-af1f-467a-9e91-6a3a16e849ce","Type":"ContainerDied","Data":"32c3e0bfa97f3fc5005b9ef54a19c1330d37f0bec040cb98ac2cfdeaba7db91e"}
Nov 03 00:26:06.936435 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.936416    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl"
Nov 03 00:26:06.936559 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.936430    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl" event={"ID":"60be433d-16ea-414a-afb7-d2e2465472b4","Type":"ContainerDied","Data":"76158754dec1cfe572438fb40c37d99d55816ea04d346b7c1648cb42318285ae"}
Nov 03 00:26:06.948684 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.948661    5012 scope.go:122] "RemoveContainer" containerID="3b86b2406a429bb58e4af58268ad82dd1a676c41d650a6816a047df8f2756dac"
Nov 03 00:26:06.962167 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.962147    5012 scope.go:122] "RemoveContainer" containerID="3b86b2406a429bb58e4af58268ad82dd1a676c41d650a6816a047df8f2756dac"
Nov 03 00:26:06.962483 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:26:06.962446    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b86b2406a429bb58e4af58268ad82dd1a676c41d650a6816a047df8f2756dac\": container with ID starting with 3b86b2406a429bb58e4af58268ad82dd1a676c41d650a6816a047df8f2756dac not found: ID does not exist" containerID="3b86b2406a429bb58e4af58268ad82dd1a676c41d650a6816a047df8f2756dac"
Nov 03 00:26:06.962582 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.962489    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b86b2406a429bb58e4af58268ad82dd1a676c41d650a6816a047df8f2756dac"} err="failed to get container status \"3b86b2406a429bb58e4af58268ad82dd1a676c41d650a6816a047df8f2756dac\": rpc error: code = NotFound desc = could not find container \"3b86b2406a429bb58e4af58268ad82dd1a676c41d650a6816a047df8f2756dac\": container with ID starting with 3b86b2406a429bb58e4af58268ad82dd1a676c41d650a6816a047df8f2756dac not found: ID does not exist"
Nov 03 00:26:06.962582 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.962505    5012 scope.go:122] "RemoveContainer" containerID="05bb992b05caffa658ceb4596319ab70c7d86074b916463a410bec4931a1880a"
Nov 03 00:26:06.987471 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.987448    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb"]
Nov 03 00:26:06.992055 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:06.992037    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-58ddf479cf-kftzb"]
Nov 03 00:26:07.021195 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:07.021162    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"]
Nov 03 00:26:07.025761 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:07.025737    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-9hwr5"]
Nov 03 00:26:07.054836 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:07.054808    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl"]
Nov 03 00:26:07.059204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:07.059182    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41e815-75758b5ff7-r6ljl"]
Nov 03 00:26:07.245505 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:07.245426    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60be433d-16ea-414a-afb7-d2e2465472b4" path="/var/lib/kubelet/pods/60be433d-16ea-414a-afb7-d2e2465472b4/volumes"
Nov 03 00:26:07.245967 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:07.245946    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e974706-7e0d-4dc1-8ccc-712a52554ef9" path="/var/lib/kubelet/pods/8e974706-7e0d-4dc1-8ccc-712a52554ef9/volumes"
Nov 03 00:26:07.246486 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:07.246465    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc956694-af1f-467a-9e91-6a3a16e849ce" path="/var/lib/kubelet/pods/dc956694-af1f-467a-9e91-6a3a16e849ce/volumes"
Nov 03 00:26:07.366320 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:26:07.366286    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:26:08.269743 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.269719    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:26:08.331803 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.331772    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-podinfo\") pod \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") "
Nov 03 00:26:08.331984 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.331832    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-credential-socket\") pod \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") "
Nov 03 00:26:08.332157 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.332136    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-token\" (UniqueName: \"kubernetes.io/projected/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-token\") pod \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") "
Nov 03 00:26:08.332332 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.332233    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-kube-api-access-qbp2x\" (UniqueName: \"kubernetes.io/projected/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-kube-api-access-qbp2x\") pod \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") "
Nov 03 00:26:08.332332 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.332140    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-credential-socket" pod "88f0a50c-b1c0-4fe7-aa15-b3050d4829f7" (UID: "88f0a50c-b1c0-4fe7-aa15-b3050d4829f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:08.332332 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.332285    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-data\") pod \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") "
Nov 03 00:26:08.332332 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.332317    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istiod-ca-cert\") pod \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") "
Nov 03 00:26:08.332531 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.332505    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-envoy\") pod \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") "
Nov 03 00:26:08.332589 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.332579    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-workload-socket\") pod \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") "
Nov 03 00:26:08.332690 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.332671    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-workload-certs\") pod \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\" (UID: \"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7\") "
Nov 03 00:26:08.332759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.332715    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-data" pod "88f0a50c-b1c0-4fe7-aa15-b3050d4829f7" (UID: "88f0a50c-b1c0-4fe7-aa15-b3050d4829f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:08.333043 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.333016    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istiod-ca-cert" pod "88f0a50c-b1c0-4fe7-aa15-b3050d4829f7" (UID: "88f0a50c-b1c0-4fe7-aa15-b3050d4829f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:26:08.333247 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.333219    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-workload-socket" pod "88f0a50c-b1c0-4fe7-aa15-b3050d4829f7" (UID: "88f0a50c-b1c0-4fe7-aa15-b3050d4829f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:08.333621 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.333454    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:08.333621 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.333480    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:08.333621 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.333494    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:08.333621 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.333506    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:08.333621 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.333567    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-workload-certs" pod "88f0a50c-b1c0-4fe7-aa15-b3050d4829f7" (UID: "88f0a50c-b1c0-4fe7-aa15-b3050d4829f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:08.334758 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.334734    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-kube-api-access-qbp2x" pod "88f0a50c-b1c0-4fe7-aa15-b3050d4829f7" (UID: "88f0a50c-b1c0-4fe7-aa15-b3050d4829f7"). InnerVolumeSpecName "kube-api-access-qbp2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:26:08.335347 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.335319    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-envoy" pod "88f0a50c-b1c0-4fe7-aa15-b3050d4829f7" (UID: "88f0a50c-b1c0-4fe7-aa15-b3050d4829f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:08.335444 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.335404    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-token" pod "88f0a50c-b1c0-4fe7-aa15-b3050d4829f7" (UID: "88f0a50c-b1c0-4fe7-aa15-b3050d4829f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:26:08.336018 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.335994    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-podinfo" pod "88f0a50c-b1c0-4fe7-aa15-b3050d4829f7" (UID: "88f0a50c-b1c0-4fe7-aa15-b3050d4829f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:26:08.435281 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.435253    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:08.435281 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.435279    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:08.435281 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.435287    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:08.435487 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.435296    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:08.435487 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.435305    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbp2x\" (UniqueName: \"kubernetes.io/projected/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7-kube-api-access-qbp2x\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:08.987088 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.987008    5012 generic.go:356] "Generic (PLEG): container finished" podID="88f0a50c-b1c0-4fe7-aa15-b3050d4829f7" containerID="f97a9eb5abe21f5fbc65159d71ff6cf4e3c2a373adbac095cdeadfb52288754c" exitCode=0
Nov 03 00:26:08.987088 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.987054    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"
Nov 03 00:26:08.987259 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.987093    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6" event={"ID":"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7","Type":"ContainerDied","Data":"f97a9eb5abe21f5fbc65159d71ff6cf4e3c2a373adbac095cdeadfb52288754c"}
Nov 03 00:26:08.987259 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.987150    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6" event={"ID":"88f0a50c-b1c0-4fe7-aa15-b3050d4829f7","Type":"ContainerDied","Data":"4cfc8918f32abb505f9cba0d5a4bbbae1b9fb0da8dedf74c23c97eec9ca178a3"}
Nov 03 00:26:08.987259 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.987176    5012 scope.go:122] "RemoveContainer" containerID="f97a9eb5abe21f5fbc65159d71ff6cf4e3c2a373adbac095cdeadfb52288754c"
Nov 03 00:26:08.999732 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:08.999708    5012 scope.go:122] "RemoveContainer" containerID="f97a9eb5abe21f5fbc65159d71ff6cf4e3c2a373adbac095cdeadfb52288754c"
Nov 03 00:26:08.999997 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:26:08.999979    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f97a9eb5abe21f5fbc65159d71ff6cf4e3c2a373adbac095cdeadfb52288754c\": container with ID starting with f97a9eb5abe21f5fbc65159d71ff6cf4e3c2a373adbac095cdeadfb52288754c not found: ID does not exist" containerID="f97a9eb5abe21f5fbc65159d71ff6cf4e3c2a373adbac095cdeadfb52288754c"
Nov 03 00:26:09.000060 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.000005    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f97a9eb5abe21f5fbc65159d71ff6cf4e3c2a373adbac095cdeadfb52288754c"} err="failed to get container status \"f97a9eb5abe21f5fbc65159d71ff6cf4e3c2a373adbac095cdeadfb52288754c\": rpc error: code = NotFound desc = could not find container \"f97a9eb5abe21f5fbc65159d71ff6cf4e3c2a373adbac095cdeadfb52288754c\": container with ID starting with f97a9eb5abe21f5fbc65159d71ff6cf4e3c2a373adbac095cdeadfb52288754c not found: ID does not exist"
Nov 03 00:26:09.036624 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.036595    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee42705a-jdn97"]
Nov 03 00:26:09.041185 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.041160    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee42705a-jdn97"]
Nov 03 00:26:09.047404 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.047384    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"]
Nov 03 00:26:09.053154 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.053132    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pf6d6"]
Nov 03 00:26:09.245511 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.245424    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17d573e6-613e-404b-b4a0-9025239a7837" path="/var/lib/kubelet/pods/17d573e6-613e-404b-b4a0-9025239a7837/volumes"
Nov 03 00:26:09.246062 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.246039    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88f0a50c-b1c0-4fe7-aa15-b3050d4829f7" path="/var/lib/kubelet/pods/88f0a50c-b1c0-4fe7-aa15-b3050d4829f7/volumes"
Nov 03 00:26:09.596921 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.596886    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c"]
Nov 03 00:26:09.597619 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.597605    5012 cpu_manager.go:414] "containerMap: removing container" podUID="dc956694-af1f-467a-9e91-6a3a16e849ce" containerName="istio-proxy"
Nov 03 00:26:09.597670 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.597620    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc956694-af1f-467a-9e91-6a3a16e849ce" containerName="istio-proxy"
Nov 03 00:26:09.597670 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.597640    5012 cpu_manager.go:414] "containerMap: removing container" podUID="88f0a50c-b1c0-4fe7-aa15-b3050d4829f7" containerName="istio-proxy"
Nov 03 00:26:09.597670 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.597646    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88f0a50c-b1c0-4fe7-aa15-b3050d4829f7" containerName="istio-proxy"
Nov 03 00:26:09.597670 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.597658    5012 cpu_manager.go:414] "containerMap: removing container" podUID="8e974706-7e0d-4dc1-8ccc-712a52554ef9" containerName="payload-pre-processing"
Nov 03 00:26:09.597670 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.597663    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e974706-7e0d-4dc1-8ccc-712a52554ef9" containerName="payload-pre-processing"
Nov 03 00:26:09.597827 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.597679    5012 cpu_manager.go:414] "containerMap: removing container" podUID="60be433d-16ea-414a-afb7-d2e2465472b4" containerName="payload-processing"
Nov 03 00:26:09.597827 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.597683    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60be433d-16ea-414a-afb7-d2e2465472b4" containerName="payload-processing"
Nov 03 00:26:09.597827 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.597785    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc956694-af1f-467a-9e91-6a3a16e849ce" containerName="istio-proxy"
Nov 03 00:26:09.597827 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.597794    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="88f0a50c-b1c0-4fe7-aa15-b3050d4829f7" containerName="istio-proxy"
Nov 03 00:26:09.597827 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.597801    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e974706-7e0d-4dc1-8ccc-712a52554ef9" containerName="payload-pre-processing"
Nov 03 00:26:09.597827 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.597808    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="60be433d-16ea-414a-afb7-d2e2465472b4" containerName="payload-processing"
Nov 03 00:26:09.603049 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.603031    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c"
Nov 03 00:26:09.617188 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.617164    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3437aa\""
Nov 03 00:26:09.620638 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.620617    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c"]
Nov 03 00:26:09.659579 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.659551    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk"]
Nov 03 00:26:09.664714 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.664689    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk"
Nov 03 00:26:09.664853 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.664714    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c8403a-423d-47ba-9417-f881d12fb749-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c\" (UID: \"90c8403a-423d-47ba-9417-f881d12fb749\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c"
Nov 03 00:26:09.664853 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.664743    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqnn4\" (UniqueName: \"kubernetes.io/projected/90c8403a-423d-47ba-9417-f881d12fb749-kube-api-access-fqnn4\") pod \"payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c\" (UID: \"90c8403a-423d-47ba-9417-f881d12fb749\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c"
Nov 03 00:26:09.690435 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.690407    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk"]
Nov 03 00:26:09.796963 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.795175    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3437aa-dockercfg-frgmz\""
Nov 03 00:26:09.839048 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.837459    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2abc571-feba-43f6-9920-5f0a0d18aa21-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk\" (UID: \"f2abc571-feba-43f6-9920-5f0a0d18aa21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk"
Nov 03 00:26:09.839048 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.837537    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c8403a-423d-47ba-9417-f881d12fb749-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c\" (UID: \"90c8403a-423d-47ba-9417-f881d12fb749\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c"
Nov 03 00:26:09.839048 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.837673    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqnn4\" (UniqueName: \"kubernetes.io/projected/90c8403a-423d-47ba-9417-f881d12fb749-kube-api-access-fqnn4\") pod \"payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c\" (UID: \"90c8403a-423d-47ba-9417-f881d12fb749\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c"
Nov 03 00:26:09.839048 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.838773    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6w5s\" (UniqueName: \"kubernetes.io/projected/f2abc571-feba-43f6-9920-5f0a0d18aa21-kube-api-access-z6w5s\") pod \"payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk\" (UID: \"f2abc571-feba-43f6-9920-5f0a0d18aa21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk"
Nov 03 00:26:09.857982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.841809    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c8403a-423d-47ba-9417-f881d12fb749-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c\" (UID: \"90c8403a-423d-47ba-9417-f881d12fb749\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c"
Nov 03 00:26:09.961839 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.958275    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2abc571-feba-43f6-9920-5f0a0d18aa21-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk\" (UID: \"f2abc571-feba-43f6-9920-5f0a0d18aa21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk"
Nov 03 00:26:09.961839 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.958537    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6w5s\" (UniqueName: \"kubernetes.io/projected/f2abc571-feba-43f6-9920-5f0a0d18aa21-kube-api-access-z6w5s\") pod \"payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk\" (UID: \"f2abc571-feba-43f6-9920-5f0a0d18aa21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk"
Nov 03 00:26:09.961839 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.961197    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2abc571-feba-43f6-9920-5f0a0d18aa21-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk\" (UID: \"f2abc571-feba-43f6-9920-5f0a0d18aa21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk"
Nov 03 00:26:09.962588 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.962375    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqnn4\" (UniqueName: \"kubernetes.io/projected/90c8403a-423d-47ba-9417-f881d12fb749-kube-api-access-fqnn4\") pod \"payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c\" (UID: \"90c8403a-423d-47ba-9417-f881d12fb749\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c"
Nov 03 00:26:09.977200 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.977127    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6w5s\" (UniqueName: \"kubernetes.io/projected/f2abc571-feba-43f6-9920-5f0a0d18aa21-kube-api-access-z6w5s\") pod \"payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk\" (UID: \"f2abc571-feba-43f6-9920-5f0a0d18aa21\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk"
Nov 03 00:26:09.980824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:09.980804    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk"
Nov 03 00:26:10.219037 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:10.218737    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c"
Nov 03 00:26:10.307117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:10.307087    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk"]
Nov 03 00:26:10.575725 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:26:10.575690    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c8403a_423d_47ba_9417_f881d12fb749.slice/crio-b67e61bed91151c6a87789e96514384717f1503ec634c6e471d6b7fdb8abe4d3 WatchSource:0}: Error finding container b67e61bed91151c6a87789e96514384717f1503ec634c6e471d6b7fdb8abe4d3: Status 404 returned error can't find the container with id b67e61bed91151c6a87789e96514384717f1503ec634c6e471d6b7fdb8abe4d3
Nov 03 00:26:10.575974 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:10.575952    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c"]
Nov 03 00:26:10.866208 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:10.866181    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655"
Nov 03 00:26:10.866647 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:10.866625    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6"
Nov 03 00:26:11.013455 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:11.013418    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c" event={"ID":"90c8403a-423d-47ba-9417-f881d12fb749","Type":"ContainerStarted","Data":"16056bdc109650940502e0f4cb8b750f8ba4b4cb30343b4ef0e54c832d2b6d2e"}
Nov 03 00:26:11.013637 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:11.013480    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c" event={"ID":"90c8403a-423d-47ba-9417-f881d12fb749","Type":"ContainerStarted","Data":"b67e61bed91151c6a87789e96514384717f1503ec634c6e471d6b7fdb8abe4d3"}
Nov 03 00:26:11.013637 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:11.013514    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c"
Nov 03 00:26:11.015106 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:11.015083    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk" event={"ID":"f2abc571-feba-43f6-9920-5f0a0d18aa21","Type":"ContainerStarted","Data":"323ba660997d36fac93777c8d21e05618b60ab9d0d234add96c0e1ace344d44e"}
Nov 03 00:26:11.015218 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:11.015124    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk" event={"ID":"f2abc571-feba-43f6-9920-5f0a0d18aa21","Type":"ContainerStarted","Data":"be826400007e30ed6a36389776a73f4c83d46e3398598e71b7764b8c6aa4cc3e"}
Nov 03 00:26:11.015218 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:11.015158    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk"
Nov 03 00:26:11.054163 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:11.054104    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c" podStartSLOduration=2.054085177 podStartE2EDuration="2.054085177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:11.048685582 +0000 UTC m=+1253.928269630" watchObservedRunningTime="2026-11-03 00:26:11.054085177 +0000 UTC m=+1253.933669222"
Nov 03 00:26:11.088289 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:11.088238    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk" podStartSLOduration=2.088224005 podStartE2EDuration="2.088224005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:11.084280272 +0000 UTC m=+1253.963864314" watchObservedRunningTime="2026-11-03 00:26:11.088224005 +0000 UTC m=+1253.967808049"
Nov 03 00:26:17.769755 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:17.769720    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"]
Nov 03 00:26:17.775190 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:17.775168    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:17.823933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:17.810960    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d43409-openshift-default-dockercfg-z468m\""
Nov 03 00:26:17.823933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:17.823056    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"]
Nov 03 00:26:18.025988 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.025057    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-workload-socket\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.025988 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.025162    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-credential-socket\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.025988 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.025306    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d99e83-8a4e-4d07-a350-571f916a764f-istio-token\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.025988 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.025338    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-workload-certs\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.025988 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.025381    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d99e83-8a4e-4d07-a350-571f916a764f-istiod-ca-cert\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.025988 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.025423    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-istio-data\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.025988 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.025467    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sckkd\" (UniqueName: \"kubernetes.io/projected/94d99e83-8a4e-4d07-a350-571f916a764f-kube-api-access-sckkd\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.025988 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.025533    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-istio-envoy\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.025988 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.025649    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d99e83-8a4e-4d07-a350-571f916a764f-istio-podinfo\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.073946 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.066247    5012 scope.go:122] "RemoveContainer" containerID="112c8521e34643047e0d702ed07444dd457f4a1dbd4671fa3c27d0e9515a8403"
Nov 03 00:26:18.175423 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.165246    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-istio-envoy\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.175423 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.165373    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d99e83-8a4e-4d07-a350-571f916a764f-istio-podinfo\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.175423 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.165552    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-workload-socket\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.175423 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.165603    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-credential-socket\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.175423 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.165715    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d99e83-8a4e-4d07-a350-571f916a764f-istio-token\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.175423 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.165743    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-workload-certs\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.175423 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.165782    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d99e83-8a4e-4d07-a350-571f916a764f-istiod-ca-cert\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.175423 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.165824    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-istio-data\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.175423 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.165865    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sckkd\" (UniqueName: \"kubernetes.io/projected/94d99e83-8a4e-4d07-a350-571f916a764f-kube-api-access-sckkd\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.175423 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.167372    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-workload-socket\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.175423 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.168129    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-credential-socket\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.175423 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.168256    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-workload-certs\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.175423 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.168964    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-istio-data\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.175423 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.169677    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d99e83-8a4e-4d07-a350-571f916a764f-istiod-ca-cert\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.176197 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.176171    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d99e83-8a4e-4d07-a350-571f916a764f-istio-podinfo\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.176492 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.176466    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-istio-envoy\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.245039 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.235772    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sckkd\" (UniqueName: \"kubernetes.io/projected/94d99e83-8a4e-4d07-a350-571f916a764f-kube-api-access-sckkd\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.249944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.249571    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d99e83-8a4e-4d07-a350-571f916a764f-istio-token\") pod \"e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") " pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.261435 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.261404    5012 scope.go:122] "RemoveContainer" containerID="17b01dfebc2fd51d74800a20fa37f2e2a27632804126da7fae17dbf2c46eb40c"
Nov 03 00:26:18.278090 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.278066    5012 scope.go:122] "RemoveContainer" containerID="1797669846dc938d1c04fb21a415cad84301ffaaf903c12db10551f45ffca9a3"
Nov 03 00:26:18.421267 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.421240    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:18.785657 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.785588    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"]
Nov 03 00:26:18.815687 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:26:18.815659    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d99e83_8a4e_4d07_a350_571f916a764f.slice/crio-aa2dd128558c7d361539f5de018c6da7c4d8eaa153f7cfdd1eaae572c1758b97 WatchSource:0}: Error finding container aa2dd128558c7d361539f5de018c6da7c4d8eaa153f7cfdd1eaae572c1758b97: Status 404 returned error can't find the container with id aa2dd128558c7d361539f5de018c6da7c4d8eaa153f7cfdd1eaae572c1758b97
Nov 03 00:26:18.818852 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.818822    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:26:18.818954 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.818890    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:26:18.818954 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:18.818935    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:26:19.216977 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:19.216037    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76" event={"ID":"94d99e83-8a4e-4d07-a350-571f916a764f","Type":"ContainerStarted","Data":"9dd6325444308fcad6adba4fbf8610961bdaec6e4291a280cbec5f1b9b731ada"}
Nov 03 00:26:19.216977 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:19.216126    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76" event={"ID":"94d99e83-8a4e-4d07-a350-571f916a764f","Type":"ContainerStarted","Data":"aa2dd128558c7d361539f5de018c6da7c4d8eaa153f7cfdd1eaae572c1758b97"}
Nov 03 00:26:19.269403 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:19.269350    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76" podStartSLOduration=2.269336572 podStartE2EDuration="2.269336572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:19.264957365 +0000 UTC m=+1262.144541409" watchObservedRunningTime="2026-11-03 00:26:19.269336572 +0000 UTC m=+1262.148920612"
Nov 03 00:26:19.425276 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:19.425242    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:19.450933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:19.450217    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:20.223243 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:20.223213    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:20.224216 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:20.224188    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:26:21.029148 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:21.029110    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c2ce0e7-p4vkh"]
Nov 03 00:26:21.042283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:21.042253    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c2ce0e7-p4vkh"]
Nov 03 00:26:21.246171 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:21.246134    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a518260-33fc-47ab-b22a-9b716fe73ae3" path="/var/lib/kubelet/pods/2a518260-33fc-47ab-b22a-9b716fe73ae3/volumes"
Nov 03 00:26:22.061126 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:22.059619    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c"
Nov 03 00:26:22.062772 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:22.062024    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk"
Nov 03 00:26:22.320091 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:22.319993    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db"]
Nov 03 00:26:22.330030 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:22.330000    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db"
Nov 03 00:26:22.352776 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:22.352363    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrtdw\" (UniqueName: \"kubernetes.io/projected/8660d374-3808-4c57-8094-32e6cdc1d73d-kube-api-access-jrtdw\") pod \"payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db\" (UID: \"8660d374-3808-4c57-8094-32e6cdc1d73d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db"
Nov 03 00:26:22.352776 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:22.352449    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db"]
Nov 03 00:26:22.352955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:22.352831    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8660d374-3808-4c57-8094-32e6cdc1d73d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db\" (UID: \"8660d374-3808-4c57-8094-32e6cdc1d73d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db"
Nov 03 00:26:22.413157 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:22.413122    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-41e815\""
Nov 03 00:26:22.413316 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:22.413229    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-41e815-dockercfg-hgspm\""
Nov 03 00:26:22.454347 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:22.454309    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8660d374-3808-4c57-8094-32e6cdc1d73d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db\" (UID: \"8660d374-3808-4c57-8094-32e6cdc1d73d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db"
Nov 03 00:26:22.454544 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:22.454428    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrtdw\" (UniqueName: \"kubernetes.io/projected/8660d374-3808-4c57-8094-32e6cdc1d73d-kube-api-access-jrtdw\") pod \"payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db\" (UID: \"8660d374-3808-4c57-8094-32e6cdc1d73d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db"
Nov 03 00:26:22.455175 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:22.455148    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8660d374-3808-4c57-8094-32e6cdc1d73d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db\" (UID: \"8660d374-3808-4c57-8094-32e6cdc1d73d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db"
Nov 03 00:26:22.472981 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:22.472952    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrtdw\" (UniqueName: \"kubernetes.io/projected/8660d374-3808-4c57-8094-32e6cdc1d73d-kube-api-access-jrtdw\") pod \"payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db\" (UID: \"8660d374-3808-4c57-8094-32e6cdc1d73d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db"
Nov 03 00:26:22.656217 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:22.656177    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db"
Nov 03 00:26:22.974853 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:26:22.974818    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8660d374_3808_4c57_8094_32e6cdc1d73d.slice/crio-1d01bb2f9b3b7b0486097b81b9c40d4bc8edf60fdaba93d86f70f5dedf18f073 WatchSource:0}: Error finding container 1d01bb2f9b3b7b0486097b81b9c40d4bc8edf60fdaba93d86f70f5dedf18f073: Status 404 returned error can't find the container with id 1d01bb2f9b3b7b0486097b81b9c40d4bc8edf60fdaba93d86f70f5dedf18f073
Nov 03 00:26:22.975021 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:22.974865    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db"]
Nov 03 00:26:23.276604 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:23.276527    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db" event={"ID":"8660d374-3808-4c57-8094-32e6cdc1d73d","Type":"ContainerStarted","Data":"1d01bb2f9b3b7b0486097b81b9c40d4bc8edf60fdaba93d86f70f5dedf18f073"}
Nov 03 00:26:23.820103 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:23.819282    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh"]
Nov 03 00:26:23.947780 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:23.947253    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh"
Nov 03 00:26:23.947780 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:23.947379    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh"]
Nov 03 00:26:23.966051 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:23.962434    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d43409\""
Nov 03 00:26:23.972718 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:23.972503    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d43409-dockercfg-pdj7r\""
Nov 03 00:26:24.021150 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:24.021114    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85ac030e-4b72-48bb-956c-90f21ca74de1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh\" (UID: \"85ac030e-4b72-48bb-956c-90f21ca74de1\") " pod="openshift-ingress/payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh"
Nov 03 00:26:24.021339 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:24.021183    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jtd5\" (UniqueName: \"kubernetes.io/projected/85ac030e-4b72-48bb-956c-90f21ca74de1-kube-api-access-9jtd5\") pod \"payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh\" (UID: \"85ac030e-4b72-48bb-956c-90f21ca74de1\") " pod="openshift-ingress/payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh"
Nov 03 00:26:24.124625 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:24.122968    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85ac030e-4b72-48bb-956c-90f21ca74de1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh\" (UID: \"85ac030e-4b72-48bb-956c-90f21ca74de1\") " pod="openshift-ingress/payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh"
Nov 03 00:26:24.124625 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:24.123021    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jtd5\" (UniqueName: \"kubernetes.io/projected/85ac030e-4b72-48bb-956c-90f21ca74de1-kube-api-access-9jtd5\") pod \"payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh\" (UID: \"85ac030e-4b72-48bb-956c-90f21ca74de1\") " pod="openshift-ingress/payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh"
Nov 03 00:26:24.124625 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:24.124559    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85ac030e-4b72-48bb-956c-90f21ca74de1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh\" (UID: \"85ac030e-4b72-48bb-956c-90f21ca74de1\") " pod="openshift-ingress/payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh"
Nov 03 00:26:24.146801 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:24.146772    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jtd5\" (UniqueName: \"kubernetes.io/projected/85ac030e-4b72-48bb-956c-90f21ca74de1-kube-api-access-9jtd5\") pod \"payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh\" (UID: \"85ac030e-4b72-48bb-956c-90f21ca74de1\") " pod="openshift-ingress/payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh"
Nov 03 00:26:24.279035 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:24.278985    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh"
Nov 03 00:26:24.607317 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:24.607287    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh"]
Nov 03 00:26:24.608284 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:26:24.608255    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ac030e_4b72_48bb_956c_90f21ca74de1.slice/crio-56d925a58048398173ddbac849f78d311b7df290952a676bc55e6494470a4644 WatchSource:0}: Error finding container 56d925a58048398173ddbac849f78d311b7df290952a676bc55e6494470a4644: Status 404 returned error can't find the container with id 56d925a58048398173ddbac849f78d311b7df290952a676bc55e6494470a4644
Nov 03 00:26:25.300348 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:25.300317    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh" event={"ID":"85ac030e-4b72-48bb-956c-90f21ca74de1","Type":"ContainerStarted","Data":"0d5b1edd934576484744ec4005578ea1bc8bdee6b4caf11ab8f39d18ad0e2556"}
Nov 03 00:26:25.300775 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:25.300374    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh" event={"ID":"85ac030e-4b72-48bb-956c-90f21ca74de1","Type":"ContainerStarted","Data":"56d925a58048398173ddbac849f78d311b7df290952a676bc55e6494470a4644"}
Nov 03 00:26:25.300775 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:25.300402    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh"
Nov 03 00:26:25.337494 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:25.337435    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh" podStartSLOduration=2.3374159629999998 podStartE2EDuration="2.337415963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:25.334552793 +0000 UTC m=+1268.214136837" watchObservedRunningTime="2026-11-03 00:26:25.337415963 +0000 UTC m=+1268.217000007"
Nov 03 00:26:26.310376 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:26.310333    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db" event={"ID":"8660d374-3808-4c57-8094-32e6cdc1d73d","Type":"ContainerStarted","Data":"e00cb6f9a92fa2b0ec8a259c2fb52528163ec0e81f9651f2584b177abf746573"}
Nov 03 00:26:26.310851 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:26.310393    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db"
Nov 03 00:26:26.312054 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:26.312029    5012 generic.go:356] "Generic (PLEG): container finished" podID="b4accff6-4a81-4c76-9b46-417203ec2921" containerID="a24155cc71d930475106728162884b1d7f972a0eea49a6ad023877410eaac831" exitCode=0
Nov 03 00:26:26.312169 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:26.312102    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4" event={"ID":"b4accff6-4a81-4c76-9b46-417203ec2921","Type":"ContainerDied","Data":"a24155cc71d930475106728162884b1d7f972a0eea49a6ad023877410eaac831"}
Nov 03 00:26:26.312533 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:26.312518    5012 scope.go:122] "RemoveContainer" containerID="a24155cc71d930475106728162884b1d7f972a0eea49a6ad023877410eaac831"
Nov 03 00:26:26.314169 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:26.314140    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:26:26.314240 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:26.314225    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:26:26.314282 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:26.314271    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:26:26.347790 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:26.347742    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db" podStartSLOduration=2.030849749 podStartE2EDuration="4.347727481s" totalImagesPullingTime="2.316877732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:26.343927877 +0000 UTC m=+1269.223511924" watchObservedRunningTime="2026-11-03 00:26:26.347727481 +0000 UTC m=+1269.227311524"
Nov 03 00:26:27.323749 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:27.323711    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4" event={"ID":"b4accff6-4a81-4c76-9b46-417203ec2921","Type":"ContainerStarted","Data":"ce4b653514fab5edac08590d4c7784ccd4eb655cfba7a723e9da3eb8e53d8dae"}
Nov 03 00:26:27.445346 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:27.445309    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:26:27.446204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:27.446178    5012 patch_prober.go:29] interesting pod/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 03 00:26:27.446338 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:27.446233    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4" podUID="b4accff6-4a81-4c76-9b46-417203ec2921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 03 00:26:28.445210 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:28.445177    5012 patch_prober.go:29] interesting pod/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 03 00:26:28.445592 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:28.445227    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4" podUID="b4accff6-4a81-4c76-9b46-417203ec2921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 03 00:26:29.445398 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:29.445365    5012 patch_prober.go:29] interesting pod/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 03 00:26:29.445882 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:29.445433    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4" podUID="b4accff6-4a81-4c76-9b46-417203ec2921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 03 00:26:29.466190 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:29.466163    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:26:30.445833 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:30.445794    5012 patch_prober.go:29] interesting pod/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 03 00:26:30.446266 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:30.445849    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4" podUID="b4accff6-4a81-4c76-9b46-417203ec2921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 03 00:26:31.028573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:31.028536    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b769d3f7-dt4q4"]
Nov 03 00:26:31.038287 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:31.038257    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b769d3f7-dt4q4"]
Nov 03 00:26:31.246311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:31.246280    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86d3e235-326b-46d9-b062-f37a3d94b661" path="/var/lib/kubelet/pods/86d3e235-326b-46d9-b062-f37a3d94b661/volumes"
Nov 03 00:26:31.451970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:31.450335    5012 patch_prober.go:29] interesting pod/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 03 00:26:31.451970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:31.450414    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4" podUID="b4accff6-4a81-4c76-9b46-417203ec2921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 03 00:26:32.445255 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:32.445221    5012 patch_prober.go:29] interesting pod/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 03 00:26:32.445419 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:32.445286    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4" podUID="b4accff6-4a81-4c76-9b46-417203ec2921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 03 00:26:33.470242 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:33.470209    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:26:33.476708 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:33.476681    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:26:36.316381 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:36.316350    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh"
Nov 03 00:26:37.326073 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:37.326045    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db"
Nov 03 00:26:37.365251 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:26:37.365216    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:26:45.276553 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.276463    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj"]
Nov 03 00:26:45.281083 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.281064    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj"
Nov 03 00:26:45.299559 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.299532    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj"]
Nov 03 00:26:45.304504 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.304483    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-dda6df\""
Nov 03 00:26:45.304615 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.304564    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-dda6df-dockercfg-pxkjr\""
Nov 03 00:26:45.344720 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.344693    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj"]
Nov 03 00:26:45.350484 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.350463    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj"
Nov 03 00:26:45.365610 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.365586    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj"]
Nov 03 00:26:45.522192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.483955    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwnq5\" (UniqueName: \"kubernetes.io/projected/c9679570-209d-48af-9805-eed8398b4895-kube-api-access-xwnq5\") pod \"payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj\" (UID: \"c9679570-209d-48af-9805-eed8398b4895\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj"
Nov 03 00:26:45.522666 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.522630    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz9x2\" (UniqueName: \"kubernetes.io/projected/75b8d3b6-0070-4826-b79d-618721119057-kube-api-access-gz9x2\") pod \"payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj\" (UID: \"75b8d3b6-0070-4826-b79d-618721119057\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj"
Nov 03 00:26:45.522961 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.522940    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75b8d3b6-0070-4826-b79d-618721119057-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj\" (UID: \"75b8d3b6-0070-4826-b79d-618721119057\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj"
Nov 03 00:26:45.523194 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.523177    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9679570-209d-48af-9805-eed8398b4895-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj\" (UID: \"c9679570-209d-48af-9805-eed8398b4895\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj"
Nov 03 00:26:45.625612 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.625571    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwnq5\" (UniqueName: \"kubernetes.io/projected/c9679570-209d-48af-9805-eed8398b4895-kube-api-access-xwnq5\") pod \"payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj\" (UID: \"c9679570-209d-48af-9805-eed8398b4895\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj"
Nov 03 00:26:45.625814 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.625651    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz9x2\" (UniqueName: \"kubernetes.io/projected/75b8d3b6-0070-4826-b79d-618721119057-kube-api-access-gz9x2\") pod \"payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj\" (UID: \"75b8d3b6-0070-4826-b79d-618721119057\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj"
Nov 03 00:26:45.625814 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.625712    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75b8d3b6-0070-4826-b79d-618721119057-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj\" (UID: \"75b8d3b6-0070-4826-b79d-618721119057\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj"
Nov 03 00:26:45.625814 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.625767    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9679570-209d-48af-9805-eed8398b4895-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj\" (UID: \"c9679570-209d-48af-9805-eed8398b4895\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj"
Nov 03 00:26:45.626760 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.626731    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75b8d3b6-0070-4826-b79d-618721119057-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj\" (UID: \"75b8d3b6-0070-4826-b79d-618721119057\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj"
Nov 03 00:26:45.626884 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.626864    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9679570-209d-48af-9805-eed8398b4895-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj\" (UID: \"c9679570-209d-48af-9805-eed8398b4895\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj"
Nov 03 00:26:45.643876 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.643841    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwnq5\" (UniqueName: \"kubernetes.io/projected/c9679570-209d-48af-9805-eed8398b4895-kube-api-access-xwnq5\") pod \"payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj\" (UID: \"c9679570-209d-48af-9805-eed8398b4895\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj"
Nov 03 00:26:45.646643 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.646604    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz9x2\" (UniqueName: \"kubernetes.io/projected/75b8d3b6-0070-4826-b79d-618721119057-kube-api-access-gz9x2\") pod \"payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj\" (UID: \"75b8d3b6-0070-4826-b79d-618721119057\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj"
Nov 03 00:26:45.647364 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.647335    5012 generic.go:356] "Generic (PLEG): container finished" podID="d88041b1-e790-4f59-aee5-a053132ef8a4" containerID="50980e2e32c08dfcdcc3371c2c8831bc59936d37d3c65d0d703fa34607169e9c" exitCode=0
Nov 03 00:26:45.647438 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.647414    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps" event={"ID":"d88041b1-e790-4f59-aee5-a053132ef8a4","Type":"ContainerDied","Data":"50980e2e32c08dfcdcc3371c2c8831bc59936d37d3c65d0d703fa34607169e9c"}
Nov 03 00:26:45.648009 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.647988    5012 scope.go:122] "RemoveContainer" containerID="50980e2e32c08dfcdcc3371c2c8831bc59936d37d3c65d0d703fa34607169e9c"
Nov 03 00:26:45.650016 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.649986    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:26:45.650121 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.650095    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:26:45.650186 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.650146    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:26:45.666615 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.666573    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj"
Nov 03 00:26:45.895145 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:45.895070    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj"
Nov 03 00:26:46.020202 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:26:46.019082    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b8d3b6_0070_4826_b79d_618721119057.slice/crio-32e82e40ab402fefdcc51b1ea8bafb89c368ae06a30e7b8376a52cd49a43dca4 WatchSource:0}: Error finding container 32e82e40ab402fefdcc51b1ea8bafb89c368ae06a30e7b8376a52cd49a43dca4: Status 404 returned error can't find the container with id 32e82e40ab402fefdcc51b1ea8bafb89c368ae06a30e7b8376a52cd49a43dca4
Nov 03 00:26:46.020202 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:46.019577    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj"]
Nov 03 00:26:46.220705 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:26:46.220670    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9679570_209d_48af_9805_eed8398b4895.slice/crio-26c2353ea2f906b4f88001c77647006bffcc45dbba23e198744f16cdd8c33964 WatchSource:0}: Error finding container 26c2353ea2f906b4f88001c77647006bffcc45dbba23e198744f16cdd8c33964: Status 404 returned error can't find the container with id 26c2353ea2f906b4f88001c77647006bffcc45dbba23e198744f16cdd8c33964
Nov 03 00:26:46.221563 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:46.221536    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj"]
Nov 03 00:26:46.662521 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:46.662476    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj" event={"ID":"c9679570-209d-48af-9805-eed8398b4895","Type":"ContainerStarted","Data":"93e454c3f86def1b02eb90e34b4922d5629b9ae508a6e8fc8beea02cd6b3fe27"}
Nov 03 00:26:46.663024 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:46.662541    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj" event={"ID":"c9679570-209d-48af-9805-eed8398b4895","Type":"ContainerStarted","Data":"26c2353ea2f906b4f88001c77647006bffcc45dbba23e198744f16cdd8c33964"}
Nov 03 00:26:46.663024 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:46.662573    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj"
Nov 03 00:26:46.664919 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:46.664874    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps" event={"ID":"d88041b1-e790-4f59-aee5-a053132ef8a4","Type":"ContainerStarted","Data":"3aa7adb14f74bf332210f1ab357dc88dee2e86a2243a3ebd7a35303281f4de95"}
Nov 03 00:26:46.666784 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:46.666755    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj" event={"ID":"75b8d3b6-0070-4826-b79d-618721119057","Type":"ContainerStarted","Data":"e39dd83175b6219a3192703190f404bababf8dde3e5de5eb2bbfc6ccab001cc7"}
Nov 03 00:26:46.666897 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:46.666810    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj"
Nov 03 00:26:46.666897 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:46.666833    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj" event={"ID":"75b8d3b6-0070-4826-b79d-618721119057","Type":"ContainerStarted","Data":"32e82e40ab402fefdcc51b1ea8bafb89c368ae06a30e7b8376a52cd49a43dca4"}
Nov 03 00:26:46.698966 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:46.698880    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj" podStartSLOduration=1.698862336 podStartE2EDuration="1.698862336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:46.697324033 +0000 UTC m=+1289.576908079" watchObservedRunningTime="2026-11-03 00:26:46.698862336 +0000 UTC m=+1289.578446381"
Nov 03 00:26:46.754940 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:46.749705    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj" podStartSLOduration=1.749686404 podStartE2EDuration="1.749686404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:46.743577896 +0000 UTC m=+1289.623161939" watchObservedRunningTime="2026-11-03 00:26:46.749686404 +0000 UTC m=+1289.629270447"
Nov 03 00:26:47.210622 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:47.207788    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:26:47.213937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:47.213093    5012 patch_prober.go:29] interesting pod/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 03 00:26:47.213937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:47.213167    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps" podUID="d88041b1-e790-4f59-aee5-a053132ef8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 03 00:26:47.667589 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:47.667557    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:26:47.678671 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:47.678641    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m"]
Nov 03 00:26:47.722659 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:47.722629    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm"]
Nov 03 00:26:48.238347 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:48.238028    5012 patch_prober.go:29] interesting pod/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 03 00:26:48.238638 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:48.238604    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps" podUID="d88041b1-e790-4f59-aee5-a053132ef8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 03 00:26:49.210995 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:49.205118    5012 patch_prober.go:29] interesting pod/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 03 00:26:49.210995 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:49.205187    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps" podUID="d88041b1-e790-4f59-aee5-a053132ef8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 03 00:26:50.205932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:50.204332    5012 patch_prober.go:29] interesting pod/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 03 00:26:50.205932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:50.204405    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps" podUID="d88041b1-e790-4f59-aee5-a053132ef8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 03 00:26:51.201946 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:51.200392    5012 patch_prober.go:29] interesting pod/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 03 00:26:51.201946 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:51.200470    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps" podUID="d88041b1-e790-4f59-aee5-a053132ef8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 03 00:26:52.211957 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:52.210396    5012 patch_prober.go:29] interesting pod/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 03 00:26:52.211957 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:52.210471    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps" podUID="d88041b1-e790-4f59-aee5-a053132ef8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 03 00:26:52.679392 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:52.679337    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m" podUID="8569e61f-5ea4-456a-a53a-adad1b4a7ae4" containerName="payload-processing" containerID="cri-o://5bccba9351fdc9c0e8527a17c28ffc09b22996c830d17b767c9c332ed4420128" gracePeriod=25
Nov 03 00:26:52.722886 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:52.722850    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm" podUID="8169d8f5-4823-4d28-ab3e-38fad6ee21bc" containerName="payload-pre-processing" containerID="cri-o://ebf1595383b22bbde529a18d2fce2b540660e627c26fe2195cfcd309d57603ad" gracePeriod=25
Nov 03 00:26:52.812292 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:52.812260    5012 generic.go:356] "Generic (PLEG): container finished" podID="8569e61f-5ea4-456a-a53a-adad1b4a7ae4" containerID="5bccba9351fdc9c0e8527a17c28ffc09b22996c830d17b767c9c332ed4420128" exitCode=0
Nov 03 00:26:52.812453 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:52.812338    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m" event={"ID":"8569e61f-5ea4-456a-a53a-adad1b4a7ae4","Type":"ContainerDied","Data":"5bccba9351fdc9c0e8527a17c28ffc09b22996c830d17b767c9c332ed4420128"}
Nov 03 00:26:53.041285 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.041258    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m"
Nov 03 00:26:53.079138 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.078242    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm"
Nov 03 00:26:53.082148 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.082120    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8569e61f-5ea4-456a-a53a-adad1b4a7ae4-kube-api-access-cxgw8\" (UniqueName: \"kubernetes.io/projected/8569e61f-5ea4-456a-a53a-adad1b4a7ae4-kube-api-access-cxgw8\") pod \"8569e61f-5ea4-456a-a53a-adad1b4a7ae4\" (UID: \"8569e61f-5ea4-456a-a53a-adad1b4a7ae4\") "
Nov 03 00:26:53.083437 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.082178    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8169d8f5-4823-4d28-ab3e-38fad6ee21bc-kube-api-access-lgzpr\" (UniqueName: \"kubernetes.io/projected/8169d8f5-4823-4d28-ab3e-38fad6ee21bc-kube-api-access-lgzpr\") pod \"8169d8f5-4823-4d28-ab3e-38fad6ee21bc\" (UID: \"8169d8f5-4823-4d28-ab3e-38fad6ee21bc\") "
Nov 03 00:26:53.083437 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.082337    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8169d8f5-4823-4d28-ab3e-38fad6ee21bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8169d8f5-4823-4d28-ab3e-38fad6ee21bc-plugins-config-volume\") pod \"8169d8f5-4823-4d28-ab3e-38fad6ee21bc\" (UID: \"8169d8f5-4823-4d28-ab3e-38fad6ee21bc\") "
Nov 03 00:26:53.083437 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.083037    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8169d8f5-4823-4d28-ab3e-38fad6ee21bc-plugins-config-volume" pod "8169d8f5-4823-4d28-ab3e-38fad6ee21bc" (UID: "8169d8f5-4823-4d28-ab3e-38fad6ee21bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:26:53.083437 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.083101    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8569e61f-5ea4-456a-a53a-adad1b4a7ae4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8569e61f-5ea4-456a-a53a-adad1b4a7ae4-plugins-config-volume\") pod \"8569e61f-5ea4-456a-a53a-adad1b4a7ae4\" (UID: \"8569e61f-5ea4-456a-a53a-adad1b4a7ae4\") "
Nov 03 00:26:53.085456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.083864    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8569e61f-5ea4-456a-a53a-adad1b4a7ae4-plugins-config-volume" pod "8569e61f-5ea4-456a-a53a-adad1b4a7ae4" (UID: "8569e61f-5ea4-456a-a53a-adad1b4a7ae4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:26:53.085456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.083971    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8169d8f5-4823-4d28-ab3e-38fad6ee21bc-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:53.087556 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.087525    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8569e61f-5ea4-456a-a53a-adad1b4a7ae4-kube-api-access-cxgw8" pod "8569e61f-5ea4-456a-a53a-adad1b4a7ae4" (UID: "8569e61f-5ea4-456a-a53a-adad1b4a7ae4"). InnerVolumeSpecName "kube-api-access-cxgw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:26:53.088609 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.088583    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8169d8f5-4823-4d28-ab3e-38fad6ee21bc-kube-api-access-lgzpr" pod "8169d8f5-4823-4d28-ab3e-38fad6ee21bc" (UID: "8169d8f5-4823-4d28-ab3e-38fad6ee21bc"). InnerVolumeSpecName "kube-api-access-lgzpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:26:53.185358 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.185325    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxgw8\" (UniqueName: \"kubernetes.io/projected/8569e61f-5ea4-456a-a53a-adad1b4a7ae4-kube-api-access-cxgw8\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:53.185358 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.185358    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgzpr\" (UniqueName: \"kubernetes.io/projected/8169d8f5-4823-4d28-ab3e-38fad6ee21bc-kube-api-access-lgzpr\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:53.185575 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.185367    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8569e61f-5ea4-456a-a53a-adad1b4a7ae4-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:26:53.203305 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.203237    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:26:53.204318 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.204299    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:26:53.829781 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.829735    5012 generic.go:356] "Generic (PLEG): container finished" podID="8169d8f5-4823-4d28-ab3e-38fad6ee21bc" containerID="ebf1595383b22bbde529a18d2fce2b540660e627c26fe2195cfcd309d57603ad" exitCode=0
Nov 03 00:26:53.845654 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.830038    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm" event={"ID":"8169d8f5-4823-4d28-ab3e-38fad6ee21bc","Type":"ContainerDied","Data":"ebf1595383b22bbde529a18d2fce2b540660e627c26fe2195cfcd309d57603ad"}
Nov 03 00:26:53.845654 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.830306    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm"
Nov 03 00:26:53.849523 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.849457    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m"
Nov 03 00:26:53.863121 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.863084    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm" event={"ID":"8169d8f5-4823-4d28-ab3e-38fad6ee21bc","Type":"ContainerDied","Data":"2316d75bba27c78a1965ee382bc802176de1d9828db9f2ea33e9bc1b07f2f0fa"}
Nov 03 00:26:53.863299 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.863152    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m" event={"ID":"8569e61f-5ea4-456a-a53a-adad1b4a7ae4","Type":"ContainerDied","Data":"3fefe7d35e3c695ee22f5f754ebfcf53cc28b3c2d502b811c1bcf59b9f4e34d9"}
Nov 03 00:26:53.863299 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.863201    5012 scope.go:122] "RemoveContainer" containerID="ebf1595383b22bbde529a18d2fce2b540660e627c26fe2195cfcd309d57603ad"
Nov 03 00:26:53.881847 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.881708    5012 scope.go:122] "RemoveContainer" containerID="ebf1595383b22bbde529a18d2fce2b540660e627c26fe2195cfcd309d57603ad"
Nov 03 00:26:53.882443 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:26:53.882301    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebf1595383b22bbde529a18d2fce2b540660e627c26fe2195cfcd309d57603ad\": container with ID starting with ebf1595383b22bbde529a18d2fce2b540660e627c26fe2195cfcd309d57603ad not found: ID does not exist" containerID="ebf1595383b22bbde529a18d2fce2b540660e627c26fe2195cfcd309d57603ad"
Nov 03 00:26:53.882443 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.882342    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf1595383b22bbde529a18d2fce2b540660e627c26fe2195cfcd309d57603ad"} err="failed to get container status \"ebf1595383b22bbde529a18d2fce2b540660e627c26fe2195cfcd309d57603ad\": rpc error: code = NotFound desc = could not find container \"ebf1595383b22bbde529a18d2fce2b540660e627c26fe2195cfcd309d57603ad\": container with ID starting with ebf1595383b22bbde529a18d2fce2b540660e627c26fe2195cfcd309d57603ad not found: ID does not exist"
Nov 03 00:26:53.882443 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.882366    5012 scope.go:122] "RemoveContainer" containerID="5bccba9351fdc9c0e8527a17c28ffc09b22996c830d17b767c9c332ed4420128"
Nov 03 00:26:53.914857 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.914829    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m"]
Nov 03 00:26:53.919576 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.919546    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f3806a-6c57db5c87-2858m"]
Nov 03 00:26:53.951817 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.951494    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm"]
Nov 03 00:26:53.958401 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:53.958365    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f3806a-5d87d748ff-7h4vm"]
Nov 03 00:26:54.752938 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:54.752896    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8"]
Nov 03 00:26:54.753672 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:54.753656    5012 cpu_manager.go:414] "containerMap: removing container" podUID="8569e61f-5ea4-456a-a53a-adad1b4a7ae4" containerName="payload-processing"
Nov 03 00:26:54.753724 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:54.753672    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8569e61f-5ea4-456a-a53a-adad1b4a7ae4" containerName="payload-processing"
Nov 03 00:26:54.753724 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:54.753693    5012 cpu_manager.go:414] "containerMap: removing container" podUID="8169d8f5-4823-4d28-ab3e-38fad6ee21bc" containerName="payload-pre-processing"
Nov 03 00:26:54.753724 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:54.753698    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8169d8f5-4823-4d28-ab3e-38fad6ee21bc" containerName="payload-pre-processing"
Nov 03 00:26:54.753862 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:54.753830    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="8569e61f-5ea4-456a-a53a-adad1b4a7ae4" containerName="payload-processing"
Nov 03 00:26:54.753862 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:54.753840    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="8169d8f5-4823-4d28-ab3e-38fad6ee21bc" containerName="payload-pre-processing"
Nov 03 00:26:54.759249 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:54.759230    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8"
Nov 03 00:26:54.775845 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:54.775817    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8"]
Nov 03 00:26:54.780062 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:54.780035    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-1d29dd-dockercfg-rqksz\""
Nov 03 00:26:54.781396 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:54.780118    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-1d29dd\""
Nov 03 00:26:54.809682 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:54.809623    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0065a2de-0be6-464b-b917-348a84615c51-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8\" (UID: \"0065a2de-0be6-464b-b917-348a84615c51\") " pod="openshift-ingress/payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8"
Nov 03 00:26:54.809845 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:54.809788    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p5hb\" (UniqueName: \"kubernetes.io/projected/0065a2de-0be6-464b-b917-348a84615c51-kube-api-access-2p5hb\") pod \"payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8\" (UID: \"0065a2de-0be6-464b-b917-348a84615c51\") " pod="openshift-ingress/payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8"
Nov 03 00:26:54.812422 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:54.812402    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2"]
Nov 03 00:26:54.817043 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:54.817026    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2"
Nov 03 00:26:54.837646 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:54.837609    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2"]
Nov 03 00:26:55.021988 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:55.010174    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0065a2de-0be6-464b-b917-348a84615c51-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8\" (UID: \"0065a2de-0be6-464b-b917-348a84615c51\") " pod="openshift-ingress/payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8"
Nov 03 00:26:55.021988 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:55.010234    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9mcl\" (UniqueName: \"kubernetes.io/projected/30042d87-468b-416c-955f-685f9b7d0994-kube-api-access-p9mcl\") pod \"payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2\" (UID: \"30042d87-468b-416c-955f-685f9b7d0994\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2"
Nov 03 00:26:55.021988 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:55.010337    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p5hb\" (UniqueName: \"kubernetes.io/projected/0065a2de-0be6-464b-b917-348a84615c51-kube-api-access-2p5hb\") pod \"payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8\" (UID: \"0065a2de-0be6-464b-b917-348a84615c51\") " pod="openshift-ingress/payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8"
Nov 03 00:26:55.021988 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:55.010460    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30042d87-468b-416c-955f-685f9b7d0994-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2\" (UID: \"30042d87-468b-416c-955f-685f9b7d0994\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2"
Nov 03 00:26:55.021988 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:55.011508    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0065a2de-0be6-464b-b917-348a84615c51-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8\" (UID: \"0065a2de-0be6-464b-b917-348a84615c51\") " pod="openshift-ingress/payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8"
Nov 03 00:26:55.076453 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:55.076424    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p5hb\" (UniqueName: \"kubernetes.io/projected/0065a2de-0be6-464b-b917-348a84615c51-kube-api-access-2p5hb\") pod \"payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8\" (UID: \"0065a2de-0be6-464b-b917-348a84615c51\") " pod="openshift-ingress/payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8"
Nov 03 00:26:55.122195 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:55.122138    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30042d87-468b-416c-955f-685f9b7d0994-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2\" (UID: \"30042d87-468b-416c-955f-685f9b7d0994\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2"
Nov 03 00:26:55.122690 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:55.122654    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9mcl\" (UniqueName: \"kubernetes.io/projected/30042d87-468b-416c-955f-685f9b7d0994-kube-api-access-p9mcl\") pod \"payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2\" (UID: \"30042d87-468b-416c-955f-685f9b7d0994\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2"
Nov 03 00:26:55.124240 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:55.124208    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30042d87-468b-416c-955f-685f9b7d0994-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2\" (UID: \"30042d87-468b-416c-955f-685f9b7d0994\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2"
Nov 03 00:26:55.143175 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:55.143136    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9mcl\" (UniqueName: \"kubernetes.io/projected/30042d87-468b-416c-955f-685f9b7d0994-kube-api-access-p9mcl\") pod \"payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2\" (UID: \"30042d87-468b-416c-955f-685f9b7d0994\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2"
Nov 03 00:26:55.246478 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:55.246433    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8169d8f5-4823-4d28-ab3e-38fad6ee21bc" path="/var/lib/kubelet/pods/8169d8f5-4823-4d28-ab3e-38fad6ee21bc/volumes"
Nov 03 00:26:55.246949 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:55.246898    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8569e61f-5ea4-456a-a53a-adad1b4a7ae4" path="/var/lib/kubelet/pods/8569e61f-5ea4-456a-a53a-adad1b4a7ae4/volumes"
Nov 03 00:26:55.377046 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:55.374503    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8"
Nov 03 00:26:55.432701 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:55.432664    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2"
Nov 03 00:26:55.703770 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:26:55.703739    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0065a2de_0be6_464b_b917_348a84615c51.slice/crio-563d7f54b57a228e30d02c888551ecd6c894e5192996fbcdf238939c329b11c2 WatchSource:0}: Error finding container 563d7f54b57a228e30d02c888551ecd6c894e5192996fbcdf238939c329b11c2: Status 404 returned error can't find the container with id 563d7f54b57a228e30d02c888551ecd6c894e5192996fbcdf238939c329b11c2
Nov 03 00:26:55.704143 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:55.704120    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8"]
Nov 03 00:26:55.781769 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:55.781744    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2"]
Nov 03 00:26:55.784465 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:26:55.784438    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30042d87_468b_416c_955f_685f9b7d0994.slice/crio-3e8cc5ba1acc16a2068c12ab1ea093ab93e7b848eae2c08c2608bd744eae7f52 WatchSource:0}: Error finding container 3e8cc5ba1acc16a2068c12ab1ea093ab93e7b848eae2c08c2608bd744eae7f52: Status 404 returned error can't find the container with id 3e8cc5ba1acc16a2068c12ab1ea093ab93e7b848eae2c08c2608bd744eae7f52
Nov 03 00:26:56.086136 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:56.086048    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8" event={"ID":"0065a2de-0be6-464b-b917-348a84615c51","Type":"ContainerStarted","Data":"a384cf5e99290841f17247aacc0845dd68543ddfc5fd631048fa76828e0f26b6"}
Nov 03 00:26:56.086136 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:56.086107    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8" event={"ID":"0065a2de-0be6-464b-b917-348a84615c51","Type":"ContainerStarted","Data":"563d7f54b57a228e30d02c888551ecd6c894e5192996fbcdf238939c329b11c2"}
Nov 03 00:26:56.086616 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:56.086158    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8"
Nov 03 00:26:56.088398 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:56.088363    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2" event={"ID":"30042d87-468b-416c-955f-685f9b7d0994","Type":"ContainerStarted","Data":"63766fe2aec56d5132efd690b59f6ca30c937f17c70f17bd0f17c515427abe37"}
Nov 03 00:26:56.088515 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:56.088415    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2" event={"ID":"30042d87-468b-416c-955f-685f9b7d0994","Type":"ContainerStarted","Data":"3e8cc5ba1acc16a2068c12ab1ea093ab93e7b848eae2c08c2608bd744eae7f52"}
Nov 03 00:26:56.088515 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:56.088459    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2"
Nov 03 00:26:56.123173 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:56.123128    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8" podStartSLOduration=2.123114539 podStartE2EDuration="2.123114539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:56.118305679 +0000 UTC m=+1298.997889724" watchObservedRunningTime="2026-11-03 00:26:56.123114539 +0000 UTC m=+1299.002698581"
Nov 03 00:26:56.156841 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:56.156797    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2" podStartSLOduration=2.156783788 podStartE2EDuration="2.156783788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:56.150965949 +0000 UTC m=+1299.030549992" watchObservedRunningTime="2026-11-03 00:26:56.156783788 +0000 UTC m=+1299.036367834"
Nov 03 00:26:57.679109 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:57.679040    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj"
Nov 03 00:26:57.679109 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:57.679104    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj"
Nov 03 00:26:58.692687 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:58.692643    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c"]
Nov 03 00:26:58.723418 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:26:58.723387    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk"]
Nov 03 00:27:02.642234 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:02.642181    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9"]
Nov 03 00:27:03.693339 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:03.693295    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c" podUID="90c8403a-423d-47ba-9417-f881d12fb749" containerName="payload-processing" containerID="cri-o://16056bdc109650940502e0f4cb8b750f8ba4b4cb30343b4ef0e54c832d2b6d2e" gracePeriod=25
Nov 03 00:27:03.724137 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:03.724092    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk" podUID="f2abc571-feba-43f6-9920-5f0a0d18aa21" containerName="payload-pre-processing" containerID="cri-o://323ba660997d36fac93777c8d21e05618b60ab9d0d234add96c0e1ace344d44e" gracePeriod=25
Nov 03 00:27:04.060899 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.060864    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c"
Nov 03 00:27:04.089371 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.089347    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk"
Nov 03 00:27:04.172411 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.172373    5012 generic.go:356] "Generic (PLEG): container finished" podID="f2abc571-feba-43f6-9920-5f0a0d18aa21" containerID="323ba660997d36fac93777c8d21e05618b60ab9d0d234add96c0e1ace344d44e" exitCode=0
Nov 03 00:27:04.172559 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.172416    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk"
Nov 03 00:27:04.172559 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.172445    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk" event={"ID":"f2abc571-feba-43f6-9920-5f0a0d18aa21","Type":"ContainerDied","Data":"323ba660997d36fac93777c8d21e05618b60ab9d0d234add96c0e1ace344d44e"}
Nov 03 00:27:04.172559 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.172525    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk" event={"ID":"f2abc571-feba-43f6-9920-5f0a0d18aa21","Type":"ContainerDied","Data":"be826400007e30ed6a36389776a73f4c83d46e3398598e71b7764b8c6aa4cc3e"}
Nov 03 00:27:04.172559 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.172554    5012 scope.go:122] "RemoveContainer" containerID="323ba660997d36fac93777c8d21e05618b60ab9d0d234add96c0e1ace344d44e"
Nov 03 00:27:04.174272 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.174239    5012 generic.go:356] "Generic (PLEG): container finished" podID="90c8403a-423d-47ba-9417-f881d12fb749" containerID="16056bdc109650940502e0f4cb8b750f8ba4b4cb30343b4ef0e54c832d2b6d2e" exitCode=0
Nov 03 00:27:04.174389 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.174288    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c"
Nov 03 00:27:04.174389 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.174292    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c" event={"ID":"90c8403a-423d-47ba-9417-f881d12fb749","Type":"ContainerDied","Data":"16056bdc109650940502e0f4cb8b750f8ba4b4cb30343b4ef0e54c832d2b6d2e"}
Nov 03 00:27:04.174389 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.174338    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c" event={"ID":"90c8403a-423d-47ba-9417-f881d12fb749","Type":"ContainerDied","Data":"b67e61bed91151c6a87789e96514384717f1503ec634c6e471d6b7fdb8abe4d3"}
Nov 03 00:27:04.187707 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.187686    5012 scope.go:122] "RemoveContainer" containerID="323ba660997d36fac93777c8d21e05618b60ab9d0d234add96c0e1ace344d44e"
Nov 03 00:27:04.188050 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:27:04.188021    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"323ba660997d36fac93777c8d21e05618b60ab9d0d234add96c0e1ace344d44e\": container with ID starting with 323ba660997d36fac93777c8d21e05618b60ab9d0d234add96c0e1ace344d44e not found: ID does not exist" containerID="323ba660997d36fac93777c8d21e05618b60ab9d0d234add96c0e1ace344d44e"
Nov 03 00:27:04.188141 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.188059    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"323ba660997d36fac93777c8d21e05618b60ab9d0d234add96c0e1ace344d44e"} err="failed to get container status \"323ba660997d36fac93777c8d21e05618b60ab9d0d234add96c0e1ace344d44e\": rpc error: code = NotFound desc = could not find container \"323ba660997d36fac93777c8d21e05618b60ab9d0d234add96c0e1ace344d44e\": container with ID starting with 323ba660997d36fac93777c8d21e05618b60ab9d0d234add96c0e1ace344d44e not found: ID does not exist"
Nov 03 00:27:04.188141 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.188081    5012 scope.go:122] "RemoveContainer" containerID="16056bdc109650940502e0f4cb8b750f8ba4b4cb30343b4ef0e54c832d2b6d2e"
Nov 03 00:27:04.199900 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.199435    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90c8403a-423d-47ba-9417-f881d12fb749-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c8403a-423d-47ba-9417-f881d12fb749-plugins-config-volume\") pod \"90c8403a-423d-47ba-9417-f881d12fb749\" (UID: \"90c8403a-423d-47ba-9417-f881d12fb749\") "
Nov 03 00:27:04.199900 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.199516    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2abc571-feba-43f6-9920-5f0a0d18aa21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2abc571-feba-43f6-9920-5f0a0d18aa21-plugins-config-volume\") pod \"f2abc571-feba-43f6-9920-5f0a0d18aa21\" (UID: \"f2abc571-feba-43f6-9920-5f0a0d18aa21\") "
Nov 03 00:27:04.199900 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.199862    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c8403a-423d-47ba-9417-f881d12fb749-plugins-config-volume" pod "90c8403a-423d-47ba-9417-f881d12fb749" (UID: "90c8403a-423d-47ba-9417-f881d12fb749"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:04.199900 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.199875    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90c8403a-423d-47ba-9417-f881d12fb749-kube-api-access-fqnn4\" (UniqueName: \"kubernetes.io/projected/90c8403a-423d-47ba-9417-f881d12fb749-kube-api-access-fqnn4\") pod \"90c8403a-423d-47ba-9417-f881d12fb749\" (UID: \"90c8403a-423d-47ba-9417-f881d12fb749\") "
Nov 03 00:27:04.200281 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.199955    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2abc571-feba-43f6-9920-5f0a0d18aa21-plugins-config-volume" pod "f2abc571-feba-43f6-9920-5f0a0d18aa21" (UID: "f2abc571-feba-43f6-9920-5f0a0d18aa21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:04.200281 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.200105    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2abc571-feba-43f6-9920-5f0a0d18aa21-kube-api-access-z6w5s\" (UniqueName: \"kubernetes.io/projected/f2abc571-feba-43f6-9920-5f0a0d18aa21-kube-api-access-z6w5s\") pod \"f2abc571-feba-43f6-9920-5f0a0d18aa21\" (UID: \"f2abc571-feba-43f6-9920-5f0a0d18aa21\") "
Nov 03 00:27:04.200281 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.200142    5012 scope.go:122] "RemoveContainer" containerID="16056bdc109650940502e0f4cb8b750f8ba4b4cb30343b4ef0e54c832d2b6d2e"
Nov 03 00:27:04.200469 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:27:04.200417    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16056bdc109650940502e0f4cb8b750f8ba4b4cb30343b4ef0e54c832d2b6d2e\": container with ID starting with 16056bdc109650940502e0f4cb8b750f8ba4b4cb30343b4ef0e54c832d2b6d2e not found: ID does not exist" containerID="16056bdc109650940502e0f4cb8b750f8ba4b4cb30343b4ef0e54c832d2b6d2e"
Nov 03 00:27:04.200469 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.200443    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16056bdc109650940502e0f4cb8b750f8ba4b4cb30343b4ef0e54c832d2b6d2e"} err="failed to get container status \"16056bdc109650940502e0f4cb8b750f8ba4b4cb30343b4ef0e54c832d2b6d2e\": rpc error: code = NotFound desc = could not find container \"16056bdc109650940502e0f4cb8b750f8ba4b4cb30343b4ef0e54c832d2b6d2e\": container with ID starting with 16056bdc109650940502e0f4cb8b750f8ba4b4cb30343b4ef0e54c832d2b6d2e not found: ID does not exist"
Nov 03 00:27:04.200687 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.200650    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c8403a-423d-47ba-9417-f881d12fb749-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:04.200687 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.200678    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2abc571-feba-43f6-9920-5f0a0d18aa21-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:04.202187 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.202162    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c8403a-423d-47ba-9417-f881d12fb749-kube-api-access-fqnn4" pod "90c8403a-423d-47ba-9417-f881d12fb749" (UID: "90c8403a-423d-47ba-9417-f881d12fb749"). InnerVolumeSpecName "kube-api-access-fqnn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:04.202347 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.202328    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2abc571-feba-43f6-9920-5f0a0d18aa21-kube-api-access-z6w5s" pod "f2abc571-feba-43f6-9920-5f0a0d18aa21" (UID: "f2abc571-feba-43f6-9920-5f0a0d18aa21"). InnerVolumeSpecName "kube-api-access-z6w5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:04.302735 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.302649    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6w5s\" (UniqueName: \"kubernetes.io/projected/f2abc571-feba-43f6-9920-5f0a0d18aa21-kube-api-access-z6w5s\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:04.302735 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.302691    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqnn4\" (UniqueName: \"kubernetes.io/projected/90c8403a-423d-47ba-9417-f881d12fb749-kube-api-access-fqnn4\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:04.361934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.361877    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"]
Nov 03 00:27:04.362248 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.362201    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5" podUID="ba16a59b-3dc5-4c8e-8378-70b27bdd0590" containerName="istio-proxy" containerID="cri-o://e5089a9404e1269ea4efbb75952ceeba291c3137ee3f0b1c6545eacac2b572d6" gracePeriod=30
Nov 03 00:27:04.369099 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.369029    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"]
Nov 03 00:27:04.369539 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.369500    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx" podUID="5338078f-b28b-4f50-9594-a7a0647050a3" containerName="istio-proxy" containerID="cri-o://63e282ca9c7b7a6f4f8c7ed1259839e0edff602d8a4439c5003a95fcc7f9a61b" gracePeriod=30
Nov 03 00:27:04.377640 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.377612    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"]
Nov 03 00:27:04.377865 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.377842    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb" podUID="76dcaa9e-a309-4329-8206-7d5ba46cc624" containerName="istio-proxy" containerID="cri-o://01a8e50be35b4e9cc90343fcc77906c8d4f72cc638f3b008ec211a1b07e84739" gracePeriod=30
Nov 03 00:27:04.535592 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.535541    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk"]
Nov 03 00:27:04.541703 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.541680    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"]
Nov 03 00:27:04.541886 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.541862    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf" podUID="fbe8c9ae-0bb1-4073-b101-8c26d16b36d0" containerName="istio-proxy" containerID="cri-o://d723e6020ac833528e82995db95681d48b4f8ebac9d9aa81440435c149c68fd9" gracePeriod=30
Nov 03 00:27:04.546623 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.546601    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"]
Nov 03 00:27:04.546806 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.546784    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh" podUID="ee103686-3d06-4f4c-ab26-111525f4b2be" containerName="istio-proxy" containerID="cri-o://f41f2f09efb258aca4732a1e1a84da6d98d913948079e4d9e25650fbc9dc744c" gracePeriod=30
Nov 03 00:27:04.551642 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.551619    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"]
Nov 03 00:27:04.551821 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.551799    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt" podUID="2ba191ff-7022-4265-a4fa-601ec997cb9b" containerName="istio-proxy" containerID="cri-o://625e3236a9daacfa6f7fe124bb8f8e0f739786f5fbc2c6273758f409ee13e311" gracePeriod=30
Nov 03 00:27:04.557037 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.556982    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"]
Nov 03 00:27:04.557190 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.557169    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl" podUID="34b0a0f0-c396-4b0d-8f77-f032a1174374" containerName="istio-proxy" containerID="cri-o://ef14ea28f3fa6621aebf8876a281d867ece48136afdb411cc6003eb9d71e5aaf" gracePeriod=30
Nov 03 00:27:04.561565 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.561540    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-7676c5b54c-ln4sk"]
Nov 03 00:27:04.584542 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.584512    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c"]
Nov 03 00:27:04.588655 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:04.588630    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5ff55ff675-tx58c"]
Nov 03 00:27:05.245184 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:05.245153    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90c8403a-423d-47ba-9417-f881d12fb749" path="/var/lib/kubelet/pods/90c8403a-423d-47ba-9417-f881d12fb749/volumes"
Nov 03 00:27:05.245606 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:05.245591    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2abc571-feba-43f6-9920-5f0a0d18aa21" path="/var/lib/kubelet/pods/f2abc571-feba-43f6-9920-5f0a0d18aa21/volumes"
Nov 03 00:27:07.099903 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:07.099872    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2"
Nov 03 00:27:07.100379 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:07.099965    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8"
Nov 03 00:27:07.365796 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:27:07.365710    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:27:07.643039 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:07.642962    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9" podUID="ebbddfa4-66d5-49ae-81c0-b8f5ae6463de" containerName="payload-pre-processing" containerID="cri-o://248b2e281840cea49897390a1375fc0a0722d5f4e822200730ed921cbf350253" gracePeriod=25
Nov 03 00:27:07.956768 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:07.956733    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9"
Nov 03 00:27:08.059041 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:08.058996    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebbddfa4-66d5-49ae-81c0-b8f5ae6463de-kube-api-access-bdccc\" (UniqueName: \"kubernetes.io/projected/ebbddfa4-66d5-49ae-81c0-b8f5ae6463de-kube-api-access-bdccc\") pod \"ebbddfa4-66d5-49ae-81c0-b8f5ae6463de\" (UID: \"ebbddfa4-66d5-49ae-81c0-b8f5ae6463de\") "
Nov 03 00:27:08.059208 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:08.059147    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebbddfa4-66d5-49ae-81c0-b8f5ae6463de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebbddfa4-66d5-49ae-81c0-b8f5ae6463de-plugins-config-volume\") pod \"ebbddfa4-66d5-49ae-81c0-b8f5ae6463de\" (UID: \"ebbddfa4-66d5-49ae-81c0-b8f5ae6463de\") "
Nov 03 00:27:08.059565 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:08.059534    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebbddfa4-66d5-49ae-81c0-b8f5ae6463de-plugins-config-volume" pod "ebbddfa4-66d5-49ae-81c0-b8f5ae6463de" (UID: "ebbddfa4-66d5-49ae-81c0-b8f5ae6463de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:08.059812 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:08.059786    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebbddfa4-66d5-49ae-81c0-b8f5ae6463de-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:08.061317 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:08.061285    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebbddfa4-66d5-49ae-81c0-b8f5ae6463de-kube-api-access-bdccc" pod "ebbddfa4-66d5-49ae-81c0-b8f5ae6463de" (UID: "ebbddfa4-66d5-49ae-81c0-b8f5ae6463de"). InnerVolumeSpecName "kube-api-access-bdccc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:08.161146 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:08.161041    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdccc\" (UniqueName: \"kubernetes.io/projected/ebbddfa4-66d5-49ae-81c0-b8f5ae6463de-kube-api-access-bdccc\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:08.222266 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:08.222226    5012 generic.go:356] "Generic (PLEG): container finished" podID="ebbddfa4-66d5-49ae-81c0-b8f5ae6463de" containerID="248b2e281840cea49897390a1375fc0a0722d5f4e822200730ed921cbf350253" exitCode=0
Nov 03 00:27:08.222497 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:08.222315    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9" event={"ID":"ebbddfa4-66d5-49ae-81c0-b8f5ae6463de","Type":"ContainerDied","Data":"248b2e281840cea49897390a1375fc0a0722d5f4e822200730ed921cbf350253"}
Nov 03 00:27:08.222497 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:08.222377    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9" event={"ID":"ebbddfa4-66d5-49ae-81c0-b8f5ae6463de","Type":"ContainerDied","Data":"4591acfeda437a1316ae42205676757f0012ffb39b09d987c6c2f6e44e2ca005"}
Nov 03 00:27:08.222497 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:08.222407    5012 scope.go:122] "RemoveContainer" containerID="248b2e281840cea49897390a1375fc0a0722d5f4e822200730ed921cbf350253"
Nov 03 00:27:08.222699 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:08.222558    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9"
Nov 03 00:27:08.238464 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:08.238444    5012 scope.go:122] "RemoveContainer" containerID="248b2e281840cea49897390a1375fc0a0722d5f4e822200730ed921cbf350253"
Nov 03 00:27:08.238789 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:27:08.238766    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"248b2e281840cea49897390a1375fc0a0722d5f4e822200730ed921cbf350253\": container with ID starting with 248b2e281840cea49897390a1375fc0a0722d5f4e822200730ed921cbf350253 not found: ID does not exist" containerID="248b2e281840cea49897390a1375fc0a0722d5f4e822200730ed921cbf350253"
Nov 03 00:27:08.238895 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:08.238793    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"248b2e281840cea49897390a1375fc0a0722d5f4e822200730ed921cbf350253"} err="failed to get container status \"248b2e281840cea49897390a1375fc0a0722d5f4e822200730ed921cbf350253\": rpc error: code = NotFound desc = could not find container \"248b2e281840cea49897390a1375fc0a0722d5f4e822200730ed921cbf350253\": container with ID starting with 248b2e281840cea49897390a1375fc0a0722d5f4e822200730ed921cbf350253 not found: ID does not exist"
Nov 03 00:27:08.273584 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:08.273555    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9"]
Nov 03 00:27:08.278242 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:08.278217    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cb05ae-67dc695747-9xwp9"]
Nov 03 00:27:09.247905 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.247869    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebbddfa4-66d5-49ae-81c0-b8f5ae6463de" path="/var/lib/kubelet/pods/ebbddfa4-66d5-49ae-81c0-b8f5ae6463de/volumes"
Nov 03 00:27:09.891738 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.891711    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:27:09.900087 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.900010    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:27:09.903509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.903439    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.991779    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-envoy\") pod \"76dcaa9e-a309-4329-8206-7d5ba46cc624\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.991845    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5338078f-b28b-4f50-9594-a7a0647050a3-kube-api-access-8czxx\" (UniqueName: \"kubernetes.io/projected/5338078f-b28b-4f50-9594-a7a0647050a3-kube-api-access-8czxx\") pod \"5338078f-b28b-4f50-9594-a7a0647050a3\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.991878    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-credential-socket\") pod \"5338078f-b28b-4f50-9594-a7a0647050a3\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.991965    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-kube-api-access-jvz2s\" (UniqueName: \"kubernetes.io/projected/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-kube-api-access-jvz2s\") pod \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.992033    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-data\") pod \"76dcaa9e-a309-4329-8206-7d5ba46cc624\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.992064    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-istio-envoy\") pod \"5338078f-b28b-4f50-9594-a7a0647050a3\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.992139    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5338078f-b28b-4f50-9594-a7a0647050a3-istio-token\" (UniqueName: \"kubernetes.io/projected/5338078f-b28b-4f50-9594-a7a0647050a3-istio-token\") pod \"5338078f-b28b-4f50-9594-a7a0647050a3\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.992187    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5338078f-b28b-4f50-9594-a7a0647050a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5338078f-b28b-4f50-9594-a7a0647050a3-istiod-ca-cert\") pod \"5338078f-b28b-4f50-9594-a7a0647050a3\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.992223    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-podinfo\") pod \"76dcaa9e-a309-4329-8206-7d5ba46cc624\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.992269    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-workload-certs\") pod \"5338078f-b28b-4f50-9594-a7a0647050a3\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.992309    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5338078f-b28b-4f50-9594-a7a0647050a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5338078f-b28b-4f50-9594-a7a0647050a3-istio-podinfo\") pod \"5338078f-b28b-4f50-9594-a7a0647050a3\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.992366    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-workload-socket\") pod \"5338078f-b28b-4f50-9594-a7a0647050a3\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.992392    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-workload-socket\") pod \"76dcaa9e-a309-4329-8206-7d5ba46cc624\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.992467    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-workload-socket\") pod \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.992516    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-credential-socket\") pod \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.992618    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-workload-certs\") pod \"76dcaa9e-a309-4329-8206-7d5ba46cc624\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.992661    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76dcaa9e-a309-4329-8206-7d5ba46cc624-kube-api-access-f8bb2\" (UniqueName: \"kubernetes.io/projected/76dcaa9e-a309-4329-8206-7d5ba46cc624-kube-api-access-f8bb2\") pod \"76dcaa9e-a309-4329-8206-7d5ba46cc624\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.992692    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-istio-data\") pod \"5338078f-b28b-4f50-9594-a7a0647050a3\" (UID: \"5338078f-b28b-4f50-9594-a7a0647050a3\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.992717    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-token\" (UniqueName: \"kubernetes.io/projected/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-token\") pod \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.992755    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istiod-ca-cert\") pod \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.992781    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76dcaa9e-a309-4329-8206-7d5ba46cc624-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76dcaa9e-a309-4329-8206-7d5ba46cc624-istiod-ca-cert\") pod \"76dcaa9e-a309-4329-8206-7d5ba46cc624\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.992824    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-podinfo\") pod \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.992878    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-credential-socket\") pod \"76dcaa9e-a309-4329-8206-7d5ba46cc624\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.992929    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-token\" (UniqueName: \"kubernetes.io/projected/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-token\") pod \"76dcaa9e-a309-4329-8206-7d5ba46cc624\" (UID: \"76dcaa9e-a309-4329-8206-7d5ba46cc624\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.992973    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-envoy\") pod \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.993000    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-data\") pod \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") "
Nov 03 00:27:09.997527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:09.993028    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-workload-certs\") pod \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\" (UID: \"ba16a59b-3dc5-4c8e-8378-70b27bdd0590\") "
Nov 03 00:27:10.003160 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.002946    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-workload-socket" pod "76dcaa9e-a309-4329-8206-7d5ba46cc624" (UID: "76dcaa9e-a309-4329-8206-7d5ba46cc624"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.003650 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.003314    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-workload-certs" pod "5338078f-b28b-4f50-9594-a7a0647050a3" (UID: "5338078f-b28b-4f50-9594-a7a0647050a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.003650 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.003419    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istiod-ca-cert" pod "ba16a59b-3dc5-4c8e-8378-70b27bdd0590" (UID: "ba16a59b-3dc5-4c8e-8378-70b27bdd0590"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:10.003811 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.003725    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-workload-socket" pod "ba16a59b-3dc5-4c8e-8378-70b27bdd0590" (UID: "ba16a59b-3dc5-4c8e-8378-70b27bdd0590"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.015164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.004038    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-credential-socket" pod "ba16a59b-3dc5-4c8e-8378-70b27bdd0590" (UID: "ba16a59b-3dc5-4c8e-8378-70b27bdd0590"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.015164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.008222    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-workload-certs" pod "ba16a59b-3dc5-4c8e-8378-70b27bdd0590" (UID: "ba16a59b-3dc5-4c8e-8378-70b27bdd0590"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.015164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.008545    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76dcaa9e-a309-4329-8206-7d5ba46cc624-istiod-ca-cert" pod "76dcaa9e-a309-4329-8206-7d5ba46cc624" (UID: "76dcaa9e-a309-4329-8206-7d5ba46cc624"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:10.015164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.009545    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.015164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.009571    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.015164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.009590    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.015164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.009605    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76dcaa9e-a309-4329-8206-7d5ba46cc624-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.015164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.009618    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.015164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.009630    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.015164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.009643    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.015164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.011535    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-credential-socket" pod "5338078f-b28b-4f50-9594-a7a0647050a3" (UID: "5338078f-b28b-4f50-9594-a7a0647050a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.015164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.011790    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-credential-socket" pod "76dcaa9e-a309-4329-8206-7d5ba46cc624" (UID: "76dcaa9e-a309-4329-8206-7d5ba46cc624"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.015164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.012420    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-istio-data" pod "5338078f-b28b-4f50-9594-a7a0647050a3" (UID: "5338078f-b28b-4f50-9594-a7a0647050a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.055961 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.053931    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-envoy" pod "76dcaa9e-a309-4329-8206-7d5ba46cc624" (UID: "76dcaa9e-a309-4329-8206-7d5ba46cc624"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.055961 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.054055    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-podinfo" pod "76dcaa9e-a309-4329-8206-7d5ba46cc624" (UID: "76dcaa9e-a309-4329-8206-7d5ba46cc624"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:27:10.055961 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.055486    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5338078f-b28b-4f50-9594-a7a0647050a3-istio-token" pod "5338078f-b28b-4f50-9594-a7a0647050a3" (UID: "5338078f-b28b-4f50-9594-a7a0647050a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:10.055961 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.055599    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-istio-envoy" pod "5338078f-b28b-4f50-9594-a7a0647050a3" (UID: "5338078f-b28b-4f50-9594-a7a0647050a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.055961 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.055656    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-podinfo" pod "ba16a59b-3dc5-4c8e-8378-70b27bdd0590" (UID: "ba16a59b-3dc5-4c8e-8378-70b27bdd0590"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:27:10.055961 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.055743    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5338078f-b28b-4f50-9594-a7a0647050a3-istio-podinfo" pod "5338078f-b28b-4f50-9594-a7a0647050a3" (UID: "5338078f-b28b-4f50-9594-a7a0647050a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:27:10.080262 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.057492    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5338078f-b28b-4f50-9594-a7a0647050a3-istiod-ca-cert" pod "5338078f-b28b-4f50-9594-a7a0647050a3" (UID: "5338078f-b28b-4f50-9594-a7a0647050a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:10.080262 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.059579    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-data" pod "ba16a59b-3dc5-4c8e-8378-70b27bdd0590" (UID: "ba16a59b-3dc5-4c8e-8378-70b27bdd0590"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.080262 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.059652    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-workload-certs" pod "76dcaa9e-a309-4329-8206-7d5ba46cc624" (UID: "76dcaa9e-a309-4329-8206-7d5ba46cc624"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.080262 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.059978    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-workload-socket" pod "5338078f-b28b-4f50-9594-a7a0647050a3" (UID: "5338078f-b28b-4f50-9594-a7a0647050a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.080262 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.060090    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76dcaa9e-a309-4329-8206-7d5ba46cc624-kube-api-access-f8bb2" pod "76dcaa9e-a309-4329-8206-7d5ba46cc624" (UID: "76dcaa9e-a309-4329-8206-7d5ba46cc624"). InnerVolumeSpecName "kube-api-access-f8bb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:10.080262 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.076237    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-kube-api-access-jvz2s" pod "ba16a59b-3dc5-4c8e-8378-70b27bdd0590" (UID: "ba16a59b-3dc5-4c8e-8378-70b27bdd0590"). InnerVolumeSpecName "kube-api-access-jvz2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:10.083249 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.081408    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-data" pod "76dcaa9e-a309-4329-8206-7d5ba46cc624" (UID: "76dcaa9e-a309-4329-8206-7d5ba46cc624"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.098549 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.098424    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:27:10.099057 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.098973    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5338078f-b28b-4f50-9594-a7a0647050a3-kube-api-access-8czxx" pod "5338078f-b28b-4f50-9594-a7a0647050a3" (UID: "5338078f-b28b-4f50-9594-a7a0647050a3"). InnerVolumeSpecName "kube-api-access-8czxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:10.099185 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.099049    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-token" pod "ba16a59b-3dc5-4c8e-8378-70b27bdd0590" (UID: "ba16a59b-3dc5-4c8e-8378-70b27bdd0590"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:10.099185 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.099098    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-token" pod "76dcaa9e-a309-4329-8206-7d5ba46cc624" (UID: "76dcaa9e-a309-4329-8206-7d5ba46cc624"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:10.104050 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.104024    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-envoy" pod "ba16a59b-3dc5-4c8e-8378-70b27bdd0590" (UID: "ba16a59b-3dc5-4c8e-8378-70b27bdd0590"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.116029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.109446    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:27:10.116029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.114878    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5338078f-b28b-4f50-9594-a7a0647050a3-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.116029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.114903    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5338078f-b28b-4f50-9594-a7a0647050a3-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.116029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.114969    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.116029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.114991    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5338078f-b28b-4f50-9594-a7a0647050a3-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.116029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.115006    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.116029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.115020    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.116029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.115033    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8bb2\" (UniqueName: \"kubernetes.io/projected/76dcaa9e-a309-4329-8206-7d5ba46cc624-kube-api-access-f8bb2\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.116029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.115047    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.116029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.115061    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.116029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.115073    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.116029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.115085    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.116029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.115098    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.116029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.115110    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.116029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.115122    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.116029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.115134    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.116029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.115146    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8czxx\" (UniqueName: \"kubernetes.io/projected/5338078f-b28b-4f50-9594-a7a0647050a3-kube-api-access-8czxx\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.116029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.115160    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.116029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.115172    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvz2s\" (UniqueName: \"kubernetes.io/projected/ba16a59b-3dc5-4c8e-8378-70b27bdd0590-kube-api-access-jvz2s\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.116029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.115186    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76dcaa9e-a309-4329-8206-7d5ba46cc624-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.116029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.115199    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5338078f-b28b-4f50-9594-a7a0647050a3-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.161694 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.161664    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:27:10.179066 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.179038    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:27:10.216674 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.216641    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-podinfo\") pod \"2ba191ff-7022-4265-a4fa-601ec997cb9b\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") "
Nov 03 00:27:10.216790 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.216721    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-envoy\") pod \"2ba191ff-7022-4265-a4fa-601ec997cb9b\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") "
Nov 03 00:27:10.216790 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.216777    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-token\" (UniqueName: \"kubernetes.io/projected/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-token\") pod \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") "
Nov 03 00:27:10.216897 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.216807    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-credential-socket\") pod \"2ba191ff-7022-4265-a4fa-601ec997cb9b\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") "
Nov 03 00:27:10.216897 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.216838    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-envoy\") pod \"34b0a0f0-c396-4b0d-8f77-f032a1174374\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") "
Nov 03 00:27:10.216897 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.216865    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-token\" (UniqueName: \"kubernetes.io/projected/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-token\") pod \"34b0a0f0-c396-4b0d-8f77-f032a1174374\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") "
Nov 03 00:27:10.217069 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.216957    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-envoy\") pod \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") "
Nov 03 00:27:10.217069 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.216993    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-data\") pod \"2ba191ff-7022-4265-a4fa-601ec997cb9b\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") "
Nov 03 00:27:10.217179 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.217043    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istiod-ca-cert\") pod \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") "
Nov 03 00:27:10.217179 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.217114    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ba191ff-7022-4265-a4fa-601ec997cb9b-kube-api-access-4vnfw\" (UniqueName: \"kubernetes.io/projected/2ba191ff-7022-4265-a4fa-601ec997cb9b-kube-api-access-4vnfw\") pod \"2ba191ff-7022-4265-a4fa-601ec997cb9b\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") "
Nov 03 00:27:10.217179 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.217147    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ba191ff-7022-4265-a4fa-601ec997cb9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba191ff-7022-4265-a4fa-601ec997cb9b-istiod-ca-cert\") pod \"2ba191ff-7022-4265-a4fa-601ec997cb9b\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") "
Nov 03 00:27:10.217179 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.217151    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-credential-socket" pod "2ba191ff-7022-4265-a4fa-601ec997cb9b" (UID: "2ba191ff-7022-4265-a4fa-601ec997cb9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.217386 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.217218    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-podinfo\") pod \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") "
Nov 03 00:27:10.217386 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.217251    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b0a0f0-c396-4b0d-8f77-f032a1174374-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b0a0f0-c396-4b0d-8f77-f032a1174374-istiod-ca-cert\") pod \"34b0a0f0-c396-4b0d-8f77-f032a1174374\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") "
Nov 03 00:27:10.217386 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.217299    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-kube-api-access-6jmg2\" (UniqueName: \"kubernetes.io/projected/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-kube-api-access-6jmg2\") pod \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") "
Nov 03 00:27:10.217386 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.217334    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-data" pod "2ba191ff-7022-4265-a4fa-601ec997cb9b" (UID: "2ba191ff-7022-4265-a4fa-601ec997cb9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.217386 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.217343    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-workload-socket\") pod \"34b0a0f0-c396-4b0d-8f77-f032a1174374\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") "
Nov 03 00:27:10.217386 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.217380    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-token\" (UniqueName: \"kubernetes.io/projected/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-token\") pod \"2ba191ff-7022-4265-a4fa-601ec997cb9b\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") "
Nov 03 00:27:10.217850 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.217414    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-workload-certs\") pod \"34b0a0f0-c396-4b0d-8f77-f032a1174374\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") "
Nov 03 00:27:10.217850 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.217453    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-credential-socket\") pod \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") "
Nov 03 00:27:10.217850 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.217536    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-data\") pod \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") "
Nov 03 00:27:10.217850 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.217550    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba191ff-7022-4265-a4fa-601ec997cb9b-istiod-ca-cert" pod "2ba191ff-7022-4265-a4fa-601ec997cb9b" (UID: "2ba191ff-7022-4265-a4fa-601ec997cb9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:10.217850 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.217581    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-workload-certs\") pod \"2ba191ff-7022-4265-a4fa-601ec997cb9b\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") "
Nov 03 00:27:10.217850 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.217615    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-podinfo\") pod \"34b0a0f0-c396-4b0d-8f77-f032a1174374\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") "
Nov 03 00:27:10.217850 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.217675    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-workload-socket\") pod \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") "
Nov 03 00:27:10.217850 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.217721    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-data\") pod \"34b0a0f0-c396-4b0d-8f77-f032a1174374\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") "
Nov 03 00:27:10.218322 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.217874    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istiod-ca-cert" pod "fbe8c9ae-0bb1-4073-b101-8c26d16b36d0" (UID: "fbe8c9ae-0bb1-4073-b101-8c26d16b36d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:10.218322 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.217894    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-workload-socket\") pod \"2ba191ff-7022-4265-a4fa-601ec997cb9b\" (UID: \"2ba191ff-7022-4265-a4fa-601ec997cb9b\") "
Nov 03 00:27:10.218322 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.217960    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b0a0f0-c396-4b0d-8f77-f032a1174374-kube-api-access-p4krc\" (UniqueName: \"kubernetes.io/projected/34b0a0f0-c396-4b0d-8f77-f032a1174374-kube-api-access-p4krc\") pod \"34b0a0f0-c396-4b0d-8f77-f032a1174374\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") "
Nov 03 00:27:10.218322 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.218022    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-credential-socket\") pod \"34b0a0f0-c396-4b0d-8f77-f032a1174374\" (UID: \"34b0a0f0-c396-4b0d-8f77-f032a1174374\") "
Nov 03 00:27:10.218322 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.218066    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-workload-certs\") pod \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\" (UID: \"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0\") "
Nov 03 00:27:10.218868 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.218843    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.219024 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.219007    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.219139 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.219126    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.219228 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.219217    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba191ff-7022-4265-a4fa-601ec997cb9b-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.219578 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.219558    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-workload-certs" pod "fbe8c9ae-0bb1-4073-b101-8c26d16b36d0" (UID: "fbe8c9ae-0bb1-4073-b101-8c26d16b36d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.221131 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.221105    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-envoy" pod "fbe8c9ae-0bb1-4073-b101-8c26d16b36d0" (UID: "fbe8c9ae-0bb1-4073-b101-8c26d16b36d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.221210 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.221144    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba191ff-7022-4265-a4fa-601ec997cb9b-kube-api-access-4vnfw" pod "2ba191ff-7022-4265-a4fa-601ec997cb9b" (UID: "2ba191ff-7022-4265-a4fa-601ec997cb9b"). InnerVolumeSpecName "kube-api-access-4vnfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:10.221361 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.221322    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-data" pod "fbe8c9ae-0bb1-4073-b101-8c26d16b36d0" (UID: "fbe8c9ae-0bb1-4073-b101-8c26d16b36d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.221414 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.221394    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-data" pod "34b0a0f0-c396-4b0d-8f77-f032a1174374" (UID: "34b0a0f0-c396-4b0d-8f77-f032a1174374"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.221632 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.221614    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-workload-certs" pod "2ba191ff-7022-4265-a4fa-601ec997cb9b" (UID: "2ba191ff-7022-4265-a4fa-601ec997cb9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.221681 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.221647    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b0a0f0-c396-4b0d-8f77-f032a1174374-istiod-ca-cert" pod "34b0a0f0-c396-4b0d-8f77-f032a1174374" (UID: "34b0a0f0-c396-4b0d-8f77-f032a1174374"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:10.221808 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.221786    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-podinfo" pod "fbe8c9ae-0bb1-4073-b101-8c26d16b36d0" (UID: "fbe8c9ae-0bb1-4073-b101-8c26d16b36d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:27:10.222074 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.222051    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-workload-socket" pod "2ba191ff-7022-4265-a4fa-601ec997cb9b" (UID: "2ba191ff-7022-4265-a4fa-601ec997cb9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.222287 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.222266    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-credential-socket" pod "34b0a0f0-c396-4b0d-8f77-f032a1174374" (UID: "34b0a0f0-c396-4b0d-8f77-f032a1174374"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.222417 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.222397    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-envoy" pod "2ba191ff-7022-4265-a4fa-601ec997cb9b" (UID: "2ba191ff-7022-4265-a4fa-601ec997cb9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.222814 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.222602    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-token" pod "fbe8c9ae-0bb1-4073-b101-8c26d16b36d0" (UID: "fbe8c9ae-0bb1-4073-b101-8c26d16b36d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:10.223175 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.222975    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-workload-socket" pod "fbe8c9ae-0bb1-4073-b101-8c26d16b36d0" (UID: "fbe8c9ae-0bb1-4073-b101-8c26d16b36d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.223175 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.223007    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-workload-socket" pod "34b0a0f0-c396-4b0d-8f77-f032a1174374" (UID: "34b0a0f0-c396-4b0d-8f77-f032a1174374"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.223175 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.223057    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-podinfo" pod "2ba191ff-7022-4265-a4fa-601ec997cb9b" (UID: "2ba191ff-7022-4265-a4fa-601ec997cb9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:27:10.223381 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.223211    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-workload-certs" pod "34b0a0f0-c396-4b0d-8f77-f032a1174374" (UID: "34b0a0f0-c396-4b0d-8f77-f032a1174374"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.223525 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.223489    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-token" pod "34b0a0f0-c396-4b0d-8f77-f032a1174374" (UID: "34b0a0f0-c396-4b0d-8f77-f032a1174374"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:10.223632 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.223596    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-credential-socket" pod "fbe8c9ae-0bb1-4073-b101-8c26d16b36d0" (UID: "fbe8c9ae-0bb1-4073-b101-8c26d16b36d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.223805 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.223778    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-podinfo" pod "34b0a0f0-c396-4b0d-8f77-f032a1174374" (UID: "34b0a0f0-c396-4b0d-8f77-f032a1174374"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:27:10.224200 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.224172    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-kube-api-access-6jmg2" pod "fbe8c9ae-0bb1-4073-b101-8c26d16b36d0" (UID: "fbe8c9ae-0bb1-4073-b101-8c26d16b36d0"). InnerVolumeSpecName "kube-api-access-6jmg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:10.224494 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.224474    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-token" pod "2ba191ff-7022-4265-a4fa-601ec997cb9b" (UID: "2ba191ff-7022-4265-a4fa-601ec997cb9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:10.224882 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.224866    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b0a0f0-c396-4b0d-8f77-f032a1174374-kube-api-access-p4krc" pod "34b0a0f0-c396-4b0d-8f77-f032a1174374" (UID: "34b0a0f0-c396-4b0d-8f77-f032a1174374"). InnerVolumeSpecName "kube-api-access-p4krc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:10.225183 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.225155    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-envoy" pod "34b0a0f0-c396-4b0d-8f77-f032a1174374" (UID: "34b0a0f0-c396-4b0d-8f77-f032a1174374"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.246879 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.246846    5012 generic.go:356] "Generic (PLEG): container finished" podID="76dcaa9e-a309-4329-8206-7d5ba46cc624" containerID="01a8e50be35b4e9cc90343fcc77906c8d4f72cc638f3b008ec211a1b07e84739" exitCode=0
Nov 03 00:27:10.247044 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.246948    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"
Nov 03 00:27:10.247044 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.246941    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb" event={"ID":"76dcaa9e-a309-4329-8206-7d5ba46cc624","Type":"ContainerDied","Data":"01a8e50be35b4e9cc90343fcc77906c8d4f72cc638f3b008ec211a1b07e84739"}
Nov 03 00:27:10.247186 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.247084    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb" event={"ID":"76dcaa9e-a309-4329-8206-7d5ba46cc624","Type":"ContainerDied","Data":"343b5399db6874fb35a276a5db599a3b468151ed10242e7515b906e8fb25fca8"}
Nov 03 00:27:10.247186 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.247119    5012 scope.go:122] "RemoveContainer" containerID="01a8e50be35b4e9cc90343fcc77906c8d4f72cc638f3b008ec211a1b07e84739"
Nov 03 00:27:10.253215 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.253187    5012 generic.go:356] "Generic (PLEG): container finished" podID="34b0a0f0-c396-4b0d-8f77-f032a1174374" containerID="ef14ea28f3fa6621aebf8876a281d867ece48136afdb411cc6003eb9d71e5aaf" exitCode=0
Nov 03 00:27:10.253729 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.253235    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"
Nov 03 00:27:10.253929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.253250    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl" event={"ID":"34b0a0f0-c396-4b0d-8f77-f032a1174374","Type":"ContainerDied","Data":"ef14ea28f3fa6621aebf8876a281d867ece48136afdb411cc6003eb9d71e5aaf"}
Nov 03 00:27:10.254069 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.254050    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl" event={"ID":"34b0a0f0-c396-4b0d-8f77-f032a1174374","Type":"ContainerDied","Data":"232098ea91feae1fd1ce51dd174eb72f493d9a2c144f3090c1e73b364d136701"}
Nov 03 00:27:10.256030 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.256003    5012 generic.go:356] "Generic (PLEG): container finished" podID="ba16a59b-3dc5-4c8e-8378-70b27bdd0590" containerID="e5089a9404e1269ea4efbb75952ceeba291c3137ee3f0b1c6545eacac2b572d6" exitCode=0
Nov 03 00:27:10.256128 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.256094    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5" event={"ID":"ba16a59b-3dc5-4c8e-8378-70b27bdd0590","Type":"ContainerDied","Data":"e5089a9404e1269ea4efbb75952ceeba291c3137ee3f0b1c6545eacac2b572d6"}
Nov 03 00:27:10.256188 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.256130    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5" event={"ID":"ba16a59b-3dc5-4c8e-8378-70b27bdd0590","Type":"ContainerDied","Data":"ce2287d07510405faca5b60fd06561f1482f851cb287cb7cbefe4e9e65ecd24f"}
Nov 03 00:27:10.256260 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.256219    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"
Nov 03 00:27:10.260431 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.260408    5012 generic.go:356] "Generic (PLEG): container finished" podID="2ba191ff-7022-4265-a4fa-601ec997cb9b" containerID="625e3236a9daacfa6f7fe124bb8f8e0f739786f5fbc2c6273758f409ee13e311" exitCode=0
Nov 03 00:27:10.260526 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.260458    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"
Nov 03 00:27:10.260526 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.260492    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt" event={"ID":"2ba191ff-7022-4265-a4fa-601ec997cb9b","Type":"ContainerDied","Data":"625e3236a9daacfa6f7fe124bb8f8e0f739786f5fbc2c6273758f409ee13e311"}
Nov 03 00:27:10.260672 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.260539    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt" event={"ID":"2ba191ff-7022-4265-a4fa-601ec997cb9b","Type":"ContainerDied","Data":"c1811678971e706937f8438cbf6828be9746760b5ac660ff8038ff73f79f5a04"}
Nov 03 00:27:10.262356 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.262183    5012 generic.go:356] "Generic (PLEG): container finished" podID="5338078f-b28b-4f50-9594-a7a0647050a3" containerID="63e282ca9c7b7a6f4f8c7ed1259839e0edff602d8a4439c5003a95fcc7f9a61b" exitCode=0
Nov 03 00:27:10.262356 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.262250    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"
Nov 03 00:27:10.262356 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.262283    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx" event={"ID":"5338078f-b28b-4f50-9594-a7a0647050a3","Type":"ContainerDied","Data":"63e282ca9c7b7a6f4f8c7ed1259839e0edff602d8a4439c5003a95fcc7f9a61b"}
Nov 03 00:27:10.262356 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.262317    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx" event={"ID":"5338078f-b28b-4f50-9594-a7a0647050a3","Type":"ContainerDied","Data":"c6e8ce2e01ab86ff04358f83ee66bc9cd2b3b7920f665ed1f09ea0800a7f5be3"}
Nov 03 00:27:10.267068 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.263855    5012 generic.go:356] "Generic (PLEG): container finished" podID="ee103686-3d06-4f4c-ab26-111525f4b2be" containerID="f41f2f09efb258aca4732a1e1a84da6d98d913948079e4d9e25650fbc9dc744c" exitCode=0
Nov 03 00:27:10.267068 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.264055    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh" event={"ID":"ee103686-3d06-4f4c-ab26-111525f4b2be","Type":"ContainerDied","Data":"f41f2f09efb258aca4732a1e1a84da6d98d913948079e4d9e25650fbc9dc744c"}
Nov 03 00:27:10.267068 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.264093    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh" event={"ID":"ee103686-3d06-4f4c-ab26-111525f4b2be","Type":"ContainerDied","Data":"ef31f5253eec973b64a9702b6e02cd15b254fc7495385ca406071908e3b5ec58"}
Nov 03 00:27:10.267068 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.264159    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"
Nov 03 00:27:10.267068 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.264299    5012 scope.go:122] "RemoveContainer" containerID="01a8e50be35b4e9cc90343fcc77906c8d4f72cc638f3b008ec211a1b07e84739"
Nov 03 00:27:10.267068 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:27:10.266269    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01a8e50be35b4e9cc90343fcc77906c8d4f72cc638f3b008ec211a1b07e84739\": container with ID starting with 01a8e50be35b4e9cc90343fcc77906c8d4f72cc638f3b008ec211a1b07e84739 not found: ID does not exist" containerID="01a8e50be35b4e9cc90343fcc77906c8d4f72cc638f3b008ec211a1b07e84739"
Nov 03 00:27:10.267068 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.266304    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01a8e50be35b4e9cc90343fcc77906c8d4f72cc638f3b008ec211a1b07e84739"} err="failed to get container status \"01a8e50be35b4e9cc90343fcc77906c8d4f72cc638f3b008ec211a1b07e84739\": rpc error: code = NotFound desc = could not find container \"01a8e50be35b4e9cc90343fcc77906c8d4f72cc638f3b008ec211a1b07e84739\": container with ID starting with 01a8e50be35b4e9cc90343fcc77906c8d4f72cc638f3b008ec211a1b07e84739 not found: ID does not exist"
Nov 03 00:27:10.267068 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.266328    5012 scope.go:122] "RemoveContainer" containerID="ef14ea28f3fa6621aebf8876a281d867ece48136afdb411cc6003eb9d71e5aaf"
Nov 03 00:27:10.271141 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.268362    5012 generic.go:356] "Generic (PLEG): container finished" podID="fbe8c9ae-0bb1-4073-b101-8c26d16b36d0" containerID="d723e6020ac833528e82995db95681d48b4f8ebac9d9aa81440435c149c68fd9" exitCode=0
Nov 03 00:27:10.271141 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.268414    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"
Nov 03 00:27:10.271141 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.268421    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf" event={"ID":"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0","Type":"ContainerDied","Data":"d723e6020ac833528e82995db95681d48b4f8ebac9d9aa81440435c149c68fd9"}
Nov 03 00:27:10.271141 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.268466    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf" event={"ID":"fbe8c9ae-0bb1-4073-b101-8c26d16b36d0","Type":"ContainerDied","Data":"3e362d6aab30ab64a1556290b63ec6aa7552ee239f7da0008c0655d5a58c5887"}
Nov 03 00:27:10.284645 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.284620    5012 scope.go:122] "RemoveContainer" containerID="ef14ea28f3fa6621aebf8876a281d867ece48136afdb411cc6003eb9d71e5aaf"
Nov 03 00:27:10.285031 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:27:10.285006    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef14ea28f3fa6621aebf8876a281d867ece48136afdb411cc6003eb9d71e5aaf\": container with ID starting with ef14ea28f3fa6621aebf8876a281d867ece48136afdb411cc6003eb9d71e5aaf not found: ID does not exist" containerID="ef14ea28f3fa6621aebf8876a281d867ece48136afdb411cc6003eb9d71e5aaf"
Nov 03 00:27:10.285200 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.285165    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef14ea28f3fa6621aebf8876a281d867ece48136afdb411cc6003eb9d71e5aaf"} err="failed to get container status \"ef14ea28f3fa6621aebf8876a281d867ece48136afdb411cc6003eb9d71e5aaf\": rpc error: code = NotFound desc = could not find container \"ef14ea28f3fa6621aebf8876a281d867ece48136afdb411cc6003eb9d71e5aaf\": container with ID starting with ef14ea28f3fa6621aebf8876a281d867ece48136afdb411cc6003eb9d71e5aaf not found: ID does not exist"
Nov 03 00:27:10.285317 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.285201    5012 scope.go:122] "RemoveContainer" containerID="e5089a9404e1269ea4efbb75952ceeba291c3137ee3f0b1c6545eacac2b572d6"
Nov 03 00:27:10.301973 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.301952    5012 scope.go:122] "RemoveContainer" containerID="e5089a9404e1269ea4efbb75952ceeba291c3137ee3f0b1c6545eacac2b572d6"
Nov 03 00:27:10.302272 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:27:10.302254    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5089a9404e1269ea4efbb75952ceeba291c3137ee3f0b1c6545eacac2b572d6\": container with ID starting with e5089a9404e1269ea4efbb75952ceeba291c3137ee3f0b1c6545eacac2b572d6 not found: ID does not exist" containerID="e5089a9404e1269ea4efbb75952ceeba291c3137ee3f0b1c6545eacac2b572d6"
Nov 03 00:27:10.302358 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.302275    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5089a9404e1269ea4efbb75952ceeba291c3137ee3f0b1c6545eacac2b572d6"} err="failed to get container status \"e5089a9404e1269ea4efbb75952ceeba291c3137ee3f0b1c6545eacac2b572d6\": rpc error: code = NotFound desc = could not find container \"e5089a9404e1269ea4efbb75952ceeba291c3137ee3f0b1c6545eacac2b572d6\": container with ID starting with e5089a9404e1269ea4efbb75952ceeba291c3137ee3f0b1c6545eacac2b572d6 not found: ID does not exist"
Nov 03 00:27:10.302358 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.302291    5012 scope.go:122] "RemoveContainer" containerID="625e3236a9daacfa6f7fe124bb8f8e0f739786f5fbc2c6273758f409ee13e311"
Nov 03 00:27:10.313740 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.313653    5012 scope.go:122] "RemoveContainer" containerID="625e3236a9daacfa6f7fe124bb8f8e0f739786f5fbc2c6273758f409ee13e311"
Nov 03 00:27:10.314022 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:27:10.313996    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"625e3236a9daacfa6f7fe124bb8f8e0f739786f5fbc2c6273758f409ee13e311\": container with ID starting with 625e3236a9daacfa6f7fe124bb8f8e0f739786f5fbc2c6273758f409ee13e311 not found: ID does not exist" containerID="625e3236a9daacfa6f7fe124bb8f8e0f739786f5fbc2c6273758f409ee13e311"
Nov 03 00:27:10.314113 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.314031    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"625e3236a9daacfa6f7fe124bb8f8e0f739786f5fbc2c6273758f409ee13e311"} err="failed to get container status \"625e3236a9daacfa6f7fe124bb8f8e0f739786f5fbc2c6273758f409ee13e311\": rpc error: code = NotFound desc = could not find container \"625e3236a9daacfa6f7fe124bb8f8e0f739786f5fbc2c6273758f409ee13e311\": container with ID starting with 625e3236a9daacfa6f7fe124bb8f8e0f739786f5fbc2c6273758f409ee13e311 not found: ID does not exist"
Nov 03 00:27:10.314113 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.314053    5012 scope.go:122] "RemoveContainer" containerID="63e282ca9c7b7a6f4f8c7ed1259839e0edff602d8a4439c5003a95fcc7f9a61b"
Nov 03 00:27:10.320462 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.320437    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-podinfo\") pod \"ee103686-3d06-4f4c-ab26-111525f4b2be\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") "
Nov 03 00:27:10.320549 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.320467    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee103686-3d06-4f4c-ab26-111525f4b2be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee103686-3d06-4f4c-ab26-111525f4b2be-istiod-ca-cert\") pod \"ee103686-3d06-4f4c-ab26-111525f4b2be\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") "
Nov 03 00:27:10.320549 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.320499    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-workload-certs\") pod \"ee103686-3d06-4f4c-ab26-111525f4b2be\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") "
Nov 03 00:27:10.320662 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.320564    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee103686-3d06-4f4c-ab26-111525f4b2be-kube-api-access-9xhrh\" (UniqueName: \"kubernetes.io/projected/ee103686-3d06-4f4c-ab26-111525f4b2be-kube-api-access-9xhrh\") pod \"ee103686-3d06-4f4c-ab26-111525f4b2be\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") "
Nov 03 00:27:10.320873 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.320846    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-workload-certs" pod "ee103686-3d06-4f4c-ab26-111525f4b2be" (UID: "ee103686-3d06-4f4c-ab26-111525f4b2be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.320873 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.320858    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee103686-3d06-4f4c-ab26-111525f4b2be-istiod-ca-cert" pod "ee103686-3d06-4f4c-ab26-111525f4b2be" (UID: "ee103686-3d06-4f4c-ab26-111525f4b2be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:10.321049 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.320638    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-envoy\") pod \"ee103686-3d06-4f4c-ab26-111525f4b2be\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") "
Nov 03 00:27:10.321106 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.321060    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-token\" (UniqueName: \"kubernetes.io/projected/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-token\") pod \"ee103686-3d06-4f4c-ab26-111525f4b2be\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") "
Nov 03 00:27:10.321106 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.321092    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-credential-socket\") pod \"ee103686-3d06-4f4c-ab26-111525f4b2be\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") "
Nov 03 00:27:10.321371 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.321327    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-workload-socket\") pod \"ee103686-3d06-4f4c-ab26-111525f4b2be\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") "
Nov 03 00:27:10.321558 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.321541    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-data\") pod \"ee103686-3d06-4f4c-ab26-111525f4b2be\" (UID: \"ee103686-3d06-4f4c-ab26-111525f4b2be\") "
Nov 03 00:27:10.321662 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.321641    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-credential-socket" pod "ee103686-3d06-4f4c-ab26-111525f4b2be" (UID: "ee103686-3d06-4f4c-ab26-111525f4b2be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.321798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.321774    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-workload-socket" pod "ee103686-3d06-4f4c-ab26-111525f4b2be" (UID: "ee103686-3d06-4f4c-ab26-111525f4b2be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.321880 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.321783    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-data" pod "ee103686-3d06-4f4c-ab26-111525f4b2be" (UID: "ee103686-3d06-4f4c-ab26-111525f4b2be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.322305 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322283    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jmg2\" (UniqueName: \"kubernetes.io/projected/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-kube-api-access-6jmg2\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.322411 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322308    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.322411 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322323    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.322411 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322336    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.322411 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322348    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.322411 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322359    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.322411 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322372    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.322411 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322385    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.322411 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322397    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.322411 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322410    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.323117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322423    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.323117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322436    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.323117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322458    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee103686-3d06-4f4c-ab26-111525f4b2be-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.323117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322471    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.323117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322483    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.323117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322497    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4krc\" (UniqueName: \"kubernetes.io/projected/34b0a0f0-c396-4b0d-8f77-f032a1174374-kube-api-access-p4krc\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.323117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322511    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.323117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322526    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.323117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322539    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.323117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322551    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba191ff-7022-4265-a4fa-601ec997cb9b-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.323117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322561    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.323117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322573    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.323117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322597    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b0a0f0-c396-4b0d-8f77-f032a1174374-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.323117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322608    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.323117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322620    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.323117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322633    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vnfw\" (UniqueName: \"kubernetes.io/projected/2ba191ff-7022-4265-a4fa-601ec997cb9b-kube-api-access-4vnfw\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.323117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322646    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.323117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.322659    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b0a0f0-c396-4b0d-8f77-f032a1174374-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.323677 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.323223    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee103686-3d06-4f4c-ab26-111525f4b2be-kube-api-access-9xhrh" pod "ee103686-3d06-4f4c-ab26-111525f4b2be" (UID: "ee103686-3d06-4f4c-ab26-111525f4b2be"). InnerVolumeSpecName "kube-api-access-9xhrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:10.323677 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.323342    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-envoy" pod "ee103686-3d06-4f4c-ab26-111525f4b2be" (UID: "ee103686-3d06-4f4c-ab26-111525f4b2be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:10.323677 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.323545    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-token" pod "ee103686-3d06-4f4c-ab26-111525f4b2be" (UID: "ee103686-3d06-4f4c-ab26-111525f4b2be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:10.324433 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.324405    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-podinfo" pod "ee103686-3d06-4f4c-ab26-111525f4b2be" (UID: "ee103686-3d06-4f4c-ab26-111525f4b2be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:27:10.325312 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.325292    5012 scope.go:122] "RemoveContainer" containerID="63e282ca9c7b7a6f4f8c7ed1259839e0edff602d8a4439c5003a95fcc7f9a61b"
Nov 03 00:27:10.325552 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:27:10.325534    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e282ca9c7b7a6f4f8c7ed1259839e0edff602d8a4439c5003a95fcc7f9a61b\": container with ID starting with 63e282ca9c7b7a6f4f8c7ed1259839e0edff602d8a4439c5003a95fcc7f9a61b not found: ID does not exist" containerID="63e282ca9c7b7a6f4f8c7ed1259839e0edff602d8a4439c5003a95fcc7f9a61b"
Nov 03 00:27:10.325600 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.325559    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e282ca9c7b7a6f4f8c7ed1259839e0edff602d8a4439c5003a95fcc7f9a61b"} err="failed to get container status \"63e282ca9c7b7a6f4f8c7ed1259839e0edff602d8a4439c5003a95fcc7f9a61b\": rpc error: code = NotFound desc = could not find container \"63e282ca9c7b7a6f4f8c7ed1259839e0edff602d8a4439c5003a95fcc7f9a61b\": container with ID starting with 63e282ca9c7b7a6f4f8c7ed1259839e0edff602d8a4439c5003a95fcc7f9a61b not found: ID does not exist"
Nov 03 00:27:10.325600 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.325574    5012 scope.go:122] "RemoveContainer" containerID="f41f2f09efb258aca4732a1e1a84da6d98d913948079e4d9e25650fbc9dc744c"
Nov 03 00:27:10.335330 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.335313    5012 scope.go:122] "RemoveContainer" containerID="f41f2f09efb258aca4732a1e1a84da6d98d913948079e4d9e25650fbc9dc744c"
Nov 03 00:27:10.335628 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:27:10.335605    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f41f2f09efb258aca4732a1e1a84da6d98d913948079e4d9e25650fbc9dc744c\": container with ID starting with f41f2f09efb258aca4732a1e1a84da6d98d913948079e4d9e25650fbc9dc744c not found: ID does not exist" containerID="f41f2f09efb258aca4732a1e1a84da6d98d913948079e4d9e25650fbc9dc744c"
Nov 03 00:27:10.335707 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.335632    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f41f2f09efb258aca4732a1e1a84da6d98d913948079e4d9e25650fbc9dc744c"} err="failed to get container status \"f41f2f09efb258aca4732a1e1a84da6d98d913948079e4d9e25650fbc9dc744c\": rpc error: code = NotFound desc = could not find container \"f41f2f09efb258aca4732a1e1a84da6d98d913948079e4d9e25650fbc9dc744c\": container with ID starting with f41f2f09efb258aca4732a1e1a84da6d98d913948079e4d9e25650fbc9dc744c not found: ID does not exist"
Nov 03 00:27:10.335707 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.335651    5012 scope.go:122] "RemoveContainer" containerID="d723e6020ac833528e82995db95681d48b4f8ebac9d9aa81440435c149c68fd9"
Nov 03 00:27:10.347348 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.347329    5012 scope.go:122] "RemoveContainer" containerID="d723e6020ac833528e82995db95681d48b4f8ebac9d9aa81440435c149c68fd9"
Nov 03 00:27:10.347594 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:27:10.347577    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d723e6020ac833528e82995db95681d48b4f8ebac9d9aa81440435c149c68fd9\": container with ID starting with d723e6020ac833528e82995db95681d48b4f8ebac9d9aa81440435c149c68fd9 not found: ID does not exist" containerID="d723e6020ac833528e82995db95681d48b4f8ebac9d9aa81440435c149c68fd9"
Nov 03 00:27:10.347648 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.347601    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d723e6020ac833528e82995db95681d48b4f8ebac9d9aa81440435c149c68fd9"} err="failed to get container status \"d723e6020ac833528e82995db95681d48b4f8ebac9d9aa81440435c149c68fd9\": rpc error: code = NotFound desc = could not find container \"d723e6020ac833528e82995db95681d48b4f8ebac9d9aa81440435c149c68fd9\": container with ID starting with d723e6020ac833528e82995db95681d48b4f8ebac9d9aa81440435c149c68fd9 not found: ID does not exist"
Nov 03 00:27:10.357741 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.357719    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"]
Nov 03 00:27:10.362359 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.362338    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-stpzb"]
Nov 03 00:27:10.408770 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.408738    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"]
Nov 03 00:27:10.415230 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.415209    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-f2kxl"]
Nov 03 00:27:10.437304 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.437277    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.437304 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.437304    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xhrh\" (UniqueName: \"kubernetes.io/projected/ee103686-3d06-4f4c-ab26-111525f4b2be-kube-api-access-9xhrh\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.437474 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.437316    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.437474 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.437327    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee103686-3d06-4f4c-ab26-111525f4b2be-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:10.450876 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.450847    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"]
Nov 03 00:27:10.455388 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.455364    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-zqczx"]
Nov 03 00:27:10.483538 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.483515    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"]
Nov 03 00:27:10.488409 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.488388    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cz7bt"]
Nov 03 00:27:10.521531 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.521509    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"]
Nov 03 00:27:10.525581 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.525562    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-l8tgf"]
Nov 03 00:27:10.553532 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.553512    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"]
Nov 03 00:27:10.559660 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.559640    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-mzdk5"]
Nov 03 00:27:10.607996 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.607975    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"]
Nov 03 00:27:10.611553 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:10.611534    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-mg7gh"]
Nov 03 00:27:11.028502 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:11.028464    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17eae966-8g62t"]
Nov 03 00:27:11.032696 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:11.032672    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17eae966-8g62t"]
Nov 03 00:27:11.244823 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:11.244792    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ba191ff-7022-4265-a4fa-601ec997cb9b" path="/var/lib/kubelet/pods/2ba191ff-7022-4265-a4fa-601ec997cb9b/volumes"
Nov 03 00:27:11.245303 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:11.245288    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b0a0f0-c396-4b0d-8f77-f032a1174374" path="/var/lib/kubelet/pods/34b0a0f0-c396-4b0d-8f77-f032a1174374/volumes"
Nov 03 00:27:11.245687 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:11.245675    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4896de11-cad4-4fda-a543-89c4cce5c068" path="/var/lib/kubelet/pods/4896de11-cad4-4fda-a543-89c4cce5c068/volumes"
Nov 03 00:27:11.246461 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:11.246434    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5338078f-b28b-4f50-9594-a7a0647050a3" path="/var/lib/kubelet/pods/5338078f-b28b-4f50-9594-a7a0647050a3/volumes"
Nov 03 00:27:11.246932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:11.246898    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76dcaa9e-a309-4329-8206-7d5ba46cc624" path="/var/lib/kubelet/pods/76dcaa9e-a309-4329-8206-7d5ba46cc624/volumes"
Nov 03 00:27:11.247372 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:11.247357    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba16a59b-3dc5-4c8e-8378-70b27bdd0590" path="/var/lib/kubelet/pods/ba16a59b-3dc5-4c8e-8378-70b27bdd0590/volumes"
Nov 03 00:27:11.248061 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:11.248047    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee103686-3d06-4f4c-ab26-111525f4b2be" path="/var/lib/kubelet/pods/ee103686-3d06-4f4c-ab26-111525f4b2be/volumes"
Nov 03 00:27:11.248472 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:11.248460    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbe8c9ae-0bb1-4073-b101-8c26d16b36d0" path="/var/lib/kubelet/pods/fbe8c9ae-0bb1-4073-b101-8c26d16b36d0/volumes"
Nov 03 00:27:17.724414 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.724379    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f"]
Nov 03 00:27:17.725169 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725152    5012 cpu_manager.go:414] "containerMap: removing container" podUID="34b0a0f0-c396-4b0d-8f77-f032a1174374" containerName="istio-proxy"
Nov 03 00:27:17.725225 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725171    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b0a0f0-c396-4b0d-8f77-f032a1174374" containerName="istio-proxy"
Nov 03 00:27:17.725225 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725188    5012 cpu_manager.go:414] "containerMap: removing container" podUID="5338078f-b28b-4f50-9594-a7a0647050a3" containerName="istio-proxy"
Nov 03 00:27:17.725225 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725193    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5338078f-b28b-4f50-9594-a7a0647050a3" containerName="istio-proxy"
Nov 03 00:27:17.725225 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725205    5012 cpu_manager.go:414] "containerMap: removing container" podUID="2ba191ff-7022-4265-a4fa-601ec997cb9b" containerName="istio-proxy"
Nov 03 00:27:17.725225 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725211    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba191ff-7022-4265-a4fa-601ec997cb9b" containerName="istio-proxy"
Nov 03 00:27:17.725225 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725217    5012 cpu_manager.go:414] "containerMap: removing container" podUID="90c8403a-423d-47ba-9417-f881d12fb749" containerName="payload-processing"
Nov 03 00:27:17.725225 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725222    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90c8403a-423d-47ba-9417-f881d12fb749" containerName="payload-processing"
Nov 03 00:27:17.725466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725240    5012 cpu_manager.go:414] "containerMap: removing container" podUID="f2abc571-feba-43f6-9920-5f0a0d18aa21" containerName="payload-pre-processing"
Nov 03 00:27:17.725466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725245    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2abc571-feba-43f6-9920-5f0a0d18aa21" containerName="payload-pre-processing"
Nov 03 00:27:17.725466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725254    5012 cpu_manager.go:414] "containerMap: removing container" podUID="ee103686-3d06-4f4c-ab26-111525f4b2be" containerName="istio-proxy"
Nov 03 00:27:17.725466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725259    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee103686-3d06-4f4c-ab26-111525f4b2be" containerName="istio-proxy"
Nov 03 00:27:17.725466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725279    5012 cpu_manager.go:414] "containerMap: removing container" podUID="fbe8c9ae-0bb1-4073-b101-8c26d16b36d0" containerName="istio-proxy"
Nov 03 00:27:17.725466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725285    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe8c9ae-0bb1-4073-b101-8c26d16b36d0" containerName="istio-proxy"
Nov 03 00:27:17.725466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725298    5012 cpu_manager.go:414] "containerMap: removing container" podUID="ebbddfa4-66d5-49ae-81c0-b8f5ae6463de" containerName="payload-pre-processing"
Nov 03 00:27:17.725466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725303    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebbddfa4-66d5-49ae-81c0-b8f5ae6463de" containerName="payload-pre-processing"
Nov 03 00:27:17.725466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725334    5012 cpu_manager.go:414] "containerMap: removing container" podUID="ba16a59b-3dc5-4c8e-8378-70b27bdd0590" containerName="istio-proxy"
Nov 03 00:27:17.725466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725339    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba16a59b-3dc5-4c8e-8378-70b27bdd0590" containerName="istio-proxy"
Nov 03 00:27:17.725466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725350    5012 cpu_manager.go:414] "containerMap: removing container" podUID="76dcaa9e-a309-4329-8206-7d5ba46cc624" containerName="istio-proxy"
Nov 03 00:27:17.725466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725355    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76dcaa9e-a309-4329-8206-7d5ba46cc624" containerName="istio-proxy"
Nov 03 00:27:17.725466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725451    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebbddfa4-66d5-49ae-81c0-b8f5ae6463de" containerName="payload-pre-processing"
Nov 03 00:27:17.725466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725464    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba16a59b-3dc5-4c8e-8378-70b27bdd0590" containerName="istio-proxy"
Nov 03 00:27:17.725466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725470    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="76dcaa9e-a309-4329-8206-7d5ba46cc624" containerName="istio-proxy"
Nov 03 00:27:17.725903 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725477    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b0a0f0-c396-4b0d-8f77-f032a1174374" containerName="istio-proxy"
Nov 03 00:27:17.725903 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725483    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="5338078f-b28b-4f50-9594-a7a0647050a3" containerName="istio-proxy"
Nov 03 00:27:17.725903 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725489    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ba191ff-7022-4265-a4fa-601ec997cb9b" containerName="istio-proxy"
Nov 03 00:27:17.725903 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725495    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="90c8403a-423d-47ba-9417-f881d12fb749" containerName="payload-processing"
Nov 03 00:27:17.725903 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725503    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2abc571-feba-43f6-9920-5f0a0d18aa21" containerName="payload-pre-processing"
Nov 03 00:27:17.725903 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725509    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee103686-3d06-4f4c-ab26-111525f4b2be" containerName="istio-proxy"
Nov 03 00:27:17.725903 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.725515    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe8c9ae-0bb1-4073-b101-8c26d16b36d0" containerName="istio-proxy"
Nov 03 00:27:17.731579 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.731556    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f"
Nov 03 00:27:17.751947 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.748285    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f"]
Nov 03 00:27:17.752879 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.752853    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 00:27:17.753424 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.753020    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 00:27:17.753424 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.753048    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1d29dd-serving-cert\""
Nov 03 00:27:17.753424 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.753083    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 00:27:17.754871 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.754850    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-x7pjw\""
Nov 03 00:27:17.843293 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.843259    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shbfs\" (UniqueName: \"kubernetes.io/projected/62381bac-7d10-4757-8d03-51b0ffe16a90-kube-api-access-shbfs\") pod \"maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f\" (UID: \"62381bac-7d10-4757-8d03-51b0ffe16a90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f"
Nov 03 00:27:17.843460 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.843310    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62381bac-7d10-4757-8d03-51b0ffe16a90-maas-api-tls\") pod \"maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f\" (UID: \"62381bac-7d10-4757-8d03-51b0ffe16a90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f"
Nov 03 00:27:17.843460 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.843433    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62381bac-7d10-4757-8d03-51b0ffe16a90-openshift-service-ca\") pod \"maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f\" (UID: \"62381bac-7d10-4757-8d03-51b0ffe16a90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f"
Nov 03 00:27:17.843627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.843612    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62381bac-7d10-4757-8d03-51b0ffe16a90-metrics-certs\") pod \"maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f\" (UID: \"62381bac-7d10-4757-8d03-51b0ffe16a90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f"
Nov 03 00:27:17.944732 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.944702    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62381bac-7d10-4757-8d03-51b0ffe16a90-metrics-certs\") pod \"maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f\" (UID: \"62381bac-7d10-4757-8d03-51b0ffe16a90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f"
Nov 03 00:27:17.944892 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.944806    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shbfs\" (UniqueName: \"kubernetes.io/projected/62381bac-7d10-4757-8d03-51b0ffe16a90-kube-api-access-shbfs\") pod \"maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f\" (UID: \"62381bac-7d10-4757-8d03-51b0ffe16a90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f"
Nov 03 00:27:17.944892 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.944831    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62381bac-7d10-4757-8d03-51b0ffe16a90-maas-api-tls\") pod \"maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f\" (UID: \"62381bac-7d10-4757-8d03-51b0ffe16a90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f"
Nov 03 00:27:17.944892 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.944874    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62381bac-7d10-4757-8d03-51b0ffe16a90-openshift-service-ca\") pod \"maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f\" (UID: \"62381bac-7d10-4757-8d03-51b0ffe16a90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f"
Nov 03 00:27:17.945625 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.945601    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62381bac-7d10-4757-8d03-51b0ffe16a90-openshift-service-ca\") pod \"maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f\" (UID: \"62381bac-7d10-4757-8d03-51b0ffe16a90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f"
Nov 03 00:27:17.947254 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.947227    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62381bac-7d10-4757-8d03-51b0ffe16a90-metrics-certs\") pod \"maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f\" (UID: \"62381bac-7d10-4757-8d03-51b0ffe16a90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f"
Nov 03 00:27:17.947254 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.947250    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62381bac-7d10-4757-8d03-51b0ffe16a90-maas-api-tls\") pod \"maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f\" (UID: \"62381bac-7d10-4757-8d03-51b0ffe16a90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f"
Nov 03 00:27:17.968799 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:17.963968    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shbfs\" (UniqueName: \"kubernetes.io/projected/62381bac-7d10-4757-8d03-51b0ffe16a90-kube-api-access-shbfs\") pod \"maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f\" (UID: \"62381bac-7d10-4757-8d03-51b0ffe16a90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f"
Nov 03 00:27:18.048051 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:18.047928    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f"
Nov 03 00:27:18.383496 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:18.383468    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f"]
Nov 03 00:27:18.383678 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:27:18.383471    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62381bac_7d10_4757_8d03_51b0ffe16a90.slice/crio-5a52ed4bc97ba7ff6e1eaa183291db0a135156fd815e5010ac5efcf893a5b3ea WatchSource:0}: Error finding container 5a52ed4bc97ba7ff6e1eaa183291db0a135156fd815e5010ac5efcf893a5b3ea: Status 404 returned error can't find the container with id 5a52ed4bc97ba7ff6e1eaa183291db0a135156fd815e5010ac5efcf893a5b3ea
Nov 03 00:27:18.385058 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:18.385042    5012 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:27:18.482793 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:18.482768    5012 scope.go:122] "RemoveContainer" containerID="386f843a0eda5dc4e88a0ffb7fca0b4d050f9e55142fb9a027e2b7b62eec4be9"
Nov 03 00:27:18.518724 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:18.518701    5012 scope.go:122] "RemoveContainer" containerID="882061f0ed159577ec5b662fcd0e074639efab3bc4e850461f38285a1f0ace66"
Nov 03 00:27:18.528666 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:18.528643    5012 scope.go:122] "RemoveContainer" containerID="4859526eb15d1ca5565f1bc6696c98980e4bb62b960b37793da09c477e962128"
Nov 03 00:27:18.538623 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:18.538603    5012 scope.go:122] "RemoveContainer" containerID="8df4439482b63bc7be24fdec7798ebb9cb86526076495b600b014c86a340d5c5"
Nov 03 00:27:18.548007 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:18.547986    5012 scope.go:122] "RemoveContainer" containerID="f36618fea66e4b39d3fb311ab8968a98461451f3a0c1eb3cc73e703c6321b551"
Nov 03 00:27:19.363006 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:19.362967    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f" event={"ID":"62381bac-7d10-4757-8d03-51b0ffe16a90","Type":"ContainerStarted","Data":"5a52ed4bc97ba7ff6e1eaa183291db0a135156fd815e5010ac5efcf893a5b3ea"}
Nov 03 00:27:19.572480 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:19.572448    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"]
Nov 03 00:27:19.572746 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:19.572702    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8" podUID="626c13b1-be66-4342-ab51-20eb70cdd9a2" containerName="istio-proxy" containerID="cri-o://298b4d5fe81e474fbdd860e7fadf671b9979e3b1112c680e1f38f8e7c35dfe96" gracePeriod=30
Nov 03 00:27:19.582239 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:19.582209    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"]
Nov 03 00:27:19.582470 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:19.582436    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57" podUID="3684a36f-b227-463d-af7f-e8472ce6a45a" containerName="istio-proxy" containerID="cri-o://b537504792fcdd667e9c372fcf794ac237e3c79f6e05bdb2ef2023df5ae32602" gracePeriod=30
Nov 03 00:27:19.627678 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:19.627551    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"]
Nov 03 00:27:19.628140 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:19.628102    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x" podUID="08b34bfd-c225-46b7-adcc-2999244af328" containerName="istio-proxy" containerID="cri-o://7115df497578842fb80ad63c681908cde635043019ae9a404fe591159d8981f4" gracePeriod=30
Nov 03 00:27:22.392444 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:22.392411    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f" event={"ID":"62381bac-7d10-4757-8d03-51b0ffe16a90","Type":"ContainerStarted","Data":"63748960e8e3ea90f82c3b136383db47399a4845b366ce101d43d6f43db7c538"}
Nov 03 00:27:22.392960 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:22.392609    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f"
Nov 03 00:27:22.430028 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:22.429979    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f" podStartSLOduration=1.76972483 podStartE2EDuration="5.429965337s" totalImagesPullingTime="3.660240507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:27:22.427450258 +0000 UTC m=+1325.307034300" watchObservedRunningTime="2026-11-03 00:27:22.429965337 +0000 UTC m=+1325.309549379"
Nov 03 00:27:24.950001 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:24.949595    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:27:25.033253 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.028984    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:27:25.054934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.035593    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:27:25.108931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.096715    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-envoy\") pod \"3684a36f-b227-463d-af7f-e8472ce6a45a\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") "
Nov 03 00:27:25.108931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.096776    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-workload-certs\") pod \"3684a36f-b227-463d-af7f-e8472ce6a45a\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") "
Nov 03 00:27:25.108931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.096813    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3684a36f-b227-463d-af7f-e8472ce6a45a-kube-api-access-vhbw9\" (UniqueName: \"kubernetes.io/projected/3684a36f-b227-463d-af7f-e8472ce6a45a-kube-api-access-vhbw9\") pod \"3684a36f-b227-463d-af7f-e8472ce6a45a\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") "
Nov 03 00:27:25.108931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.096880    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3684a36f-b227-463d-af7f-e8472ce6a45a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3684a36f-b227-463d-af7f-e8472ce6a45a-istiod-ca-cert\") pod \"3684a36f-b227-463d-af7f-e8472ce6a45a\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") "
Nov 03 00:27:25.108931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.097058    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-workload-socket\") pod \"3684a36f-b227-463d-af7f-e8472ce6a45a\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") "
Nov 03 00:27:25.108931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.097088    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-credential-socket\") pod \"3684a36f-b227-463d-af7f-e8472ce6a45a\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") "
Nov 03 00:27:25.108931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.097123    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-podinfo\") pod \"3684a36f-b227-463d-af7f-e8472ce6a45a\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") "
Nov 03 00:27:25.108931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.097167    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-data\") pod \"3684a36f-b227-463d-af7f-e8472ce6a45a\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") "
Nov 03 00:27:25.108931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.097217    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-token\" (UniqueName: \"kubernetes.io/projected/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-token\") pod \"3684a36f-b227-463d-af7f-e8472ce6a45a\" (UID: \"3684a36f-b227-463d-af7f-e8472ce6a45a\") "
Nov 03 00:27:25.121329 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.120625    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3684a36f-b227-463d-af7f-e8472ce6a45a-istiod-ca-cert" pod "3684a36f-b227-463d-af7f-e8472ce6a45a" (UID: "3684a36f-b227-463d-af7f-e8472ce6a45a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:25.122241 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.122211    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-workload-socket" pod "3684a36f-b227-463d-af7f-e8472ce6a45a" (UID: "3684a36f-b227-463d-af7f-e8472ce6a45a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:25.122518 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.122462    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-data" pod "3684a36f-b227-463d-af7f-e8472ce6a45a" (UID: "3684a36f-b227-463d-af7f-e8472ce6a45a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:25.125205 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.125174    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-podinfo" pod "3684a36f-b227-463d-af7f-e8472ce6a45a" (UID: "3684a36f-b227-463d-af7f-e8472ce6a45a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:27:25.136008 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.134972    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-workload-certs" pod "3684a36f-b227-463d-af7f-e8472ce6a45a" (UID: "3684a36f-b227-463d-af7f-e8472ce6a45a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:25.139228 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.139181    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-envoy" pod "3684a36f-b227-463d-af7f-e8472ce6a45a" (UID: "3684a36f-b227-463d-af7f-e8472ce6a45a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:25.166129 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.166066    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-credential-socket" pod "3684a36f-b227-463d-af7f-e8472ce6a45a" (UID: "3684a36f-b227-463d-af7f-e8472ce6a45a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:25.181994 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.181957    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67"]
Nov 03 00:27:25.183803 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.183169    5012 cpu_manager.go:414] "containerMap: removing container" podUID="626c13b1-be66-4342-ab51-20eb70cdd9a2" containerName="istio-proxy"
Nov 03 00:27:25.183803 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.183195    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="626c13b1-be66-4342-ab51-20eb70cdd9a2" containerName="istio-proxy"
Nov 03 00:27:25.183803 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.183216    5012 cpu_manager.go:414] "containerMap: removing container" podUID="3684a36f-b227-463d-af7f-e8472ce6a45a" containerName="istio-proxy"
Nov 03 00:27:25.183803 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.183226    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3684a36f-b227-463d-af7f-e8472ce6a45a" containerName="istio-proxy"
Nov 03 00:27:25.183803 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.183301    5012 cpu_manager.go:414] "containerMap: removing container" podUID="08b34bfd-c225-46b7-adcc-2999244af328" containerName="istio-proxy"
Nov 03 00:27:25.183803 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.183309    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08b34bfd-c225-46b7-adcc-2999244af328" containerName="istio-proxy"
Nov 03 00:27:25.183803 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.183624    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="08b34bfd-c225-46b7-adcc-2999244af328" containerName="istio-proxy"
Nov 03 00:27:25.183803 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.183648    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="626c13b1-be66-4342-ab51-20eb70cdd9a2" containerName="istio-proxy"
Nov 03 00:27:25.183803 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.183663    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="3684a36f-b227-463d-af7f-e8472ce6a45a" containerName="istio-proxy"
Nov 03 00:27:25.185336 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.185287    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3684a36f-b227-463d-af7f-e8472ce6a45a-kube-api-access-vhbw9" pod "3684a36f-b227-463d-af7f-e8472ce6a45a" (UID: "3684a36f-b227-463d-af7f-e8472ce6a45a"). InnerVolumeSpecName "kube-api-access-vhbw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:25.186024 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.185995    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-token" pod "3684a36f-b227-463d-af7f-e8472ce6a45a" (UID: "3684a36f-b227-463d-af7f-e8472ce6a45a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:25.189154 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.188522    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67"
Nov 03 00:27:25.193399 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.193326    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67"]
Nov 03 00:27:25.203201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.203051    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3437aa\""
Nov 03 00:27:25.203201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.203174    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3437aa-dockercfg-sk7xj\""
Nov 03 00:27:25.203375 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.203245    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-credential-socket\") pod \"08b34bfd-c225-46b7-adcc-2999244af328\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") "
Nov 03 00:27:25.203375 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.203289    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-podinfo\") pod \"626c13b1-be66-4342-ab51-20eb70cdd9a2\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") "
Nov 03 00:27:25.203375 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.203324    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-istio-data\") pod \"08b34bfd-c225-46b7-adcc-2999244af328\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") "
Nov 03 00:27:25.203524 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.203448    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08b34bfd-c225-46b7-adcc-2999244af328-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b34bfd-c225-46b7-adcc-2999244af328-istio-podinfo\") pod \"08b34bfd-c225-46b7-adcc-2999244af328\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") "
Nov 03 00:27:25.203524 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.203512    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-data\") pod \"626c13b1-be66-4342-ab51-20eb70cdd9a2\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") "
Nov 03 00:27:25.203631 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.203553    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-token\" (UniqueName: \"kubernetes.io/projected/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-token\") pod \"626c13b1-be66-4342-ab51-20eb70cdd9a2\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") "
Nov 03 00:27:25.203631 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.203617    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08b34bfd-c225-46b7-adcc-2999244af328-istio-token\" (UniqueName: \"kubernetes.io/projected/08b34bfd-c225-46b7-adcc-2999244af328-istio-token\") pod \"08b34bfd-c225-46b7-adcc-2999244af328\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") "
Nov 03 00:27:25.203733 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.203671    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-workload-socket\") pod \"08b34bfd-c225-46b7-adcc-2999244af328\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") "
Nov 03 00:27:25.203733 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.203721    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-credential-socket\") pod \"626c13b1-be66-4342-ab51-20eb70cdd9a2\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") "
Nov 03 00:27:25.203837 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.203758    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-workload-certs\") pod \"626c13b1-be66-4342-ab51-20eb70cdd9a2\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") "
Nov 03 00:27:25.203837 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.203802    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-istio-data" pod "08b34bfd-c225-46b7-adcc-2999244af328" (UID: "08b34bfd-c225-46b7-adcc-2999244af328"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:25.203837 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.203832    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-istio-envoy\") pod \"08b34bfd-c225-46b7-adcc-2999244af328\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") "
Nov 03 00:27:25.204011 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.203866    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-workload-socket\") pod \"626c13b1-be66-4342-ab51-20eb70cdd9a2\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") "
Nov 03 00:27:25.204011 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.203924    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/626c13b1-be66-4342-ab51-20eb70cdd9a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/626c13b1-be66-4342-ab51-20eb70cdd9a2-istiod-ca-cert\") pod \"626c13b1-be66-4342-ab51-20eb70cdd9a2\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") "
Nov 03 00:27:25.204011 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.203965    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08b34bfd-c225-46b7-adcc-2999244af328-kube-api-access-zgdr8\" (UniqueName: \"kubernetes.io/projected/08b34bfd-c225-46b7-adcc-2999244af328-kube-api-access-zgdr8\") pod \"08b34bfd-c225-46b7-adcc-2999244af328\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") "
Nov 03 00:27:25.204169 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.204092    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-credential-socket" pod "08b34bfd-c225-46b7-adcc-2999244af328" (UID: "08b34bfd-c225-46b7-adcc-2999244af328"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:25.204789 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.204270    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-workload-socket" pod "08b34bfd-c225-46b7-adcc-2999244af328" (UID: "08b34bfd-c225-46b7-adcc-2999244af328"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:25.204789 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.204532    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/626c13b1-be66-4342-ab51-20eb70cdd9a2-kube-api-access-w4vzj\" (UniqueName: \"kubernetes.io/projected/626c13b1-be66-4342-ab51-20eb70cdd9a2-kube-api-access-w4vzj\") pod \"626c13b1-be66-4342-ab51-20eb70cdd9a2\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") "
Nov 03 00:27:25.204789 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.204567    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-workload-certs\") pod \"08b34bfd-c225-46b7-adcc-2999244af328\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") "
Nov 03 00:27:25.204789 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.204634    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08b34bfd-c225-46b7-adcc-2999244af328-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b34bfd-c225-46b7-adcc-2999244af328-istiod-ca-cert\") pod \"08b34bfd-c225-46b7-adcc-2999244af328\" (UID: \"08b34bfd-c225-46b7-adcc-2999244af328\") "
Nov 03 00:27:25.204789 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.204665    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-envoy\") pod \"626c13b1-be66-4342-ab51-20eb70cdd9a2\" (UID: \"626c13b1-be66-4342-ab51-20eb70cdd9a2\") "
Nov 03 00:27:25.205982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.205404    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.205982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.205431    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.205982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.205445    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhbw9\" (UniqueName: \"kubernetes.io/projected/3684a36f-b227-463d-af7f-e8472ce6a45a-kube-api-access-vhbw9\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.205982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.205459    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3684a36f-b227-463d-af7f-e8472ce6a45a-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.205982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.205473    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.205982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.205486    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.205982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.205500    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.205982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.205515    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.205982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.205528    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.205982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.205542    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.205982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.205554    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.205982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.205568    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3684a36f-b227-463d-af7f-e8472ce6a45a-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.212974 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.207203    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-data" pod "626c13b1-be66-4342-ab51-20eb70cdd9a2" (UID: "626c13b1-be66-4342-ab51-20eb70cdd9a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:25.212974 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.211715    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b34bfd-c225-46b7-adcc-2999244af328-kube-api-access-zgdr8" pod "08b34bfd-c225-46b7-adcc-2999244af328" (UID: "08b34bfd-c225-46b7-adcc-2999244af328"). InnerVolumeSpecName "kube-api-access-zgdr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:25.212974 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.211979    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-workload-socket" pod "626c13b1-be66-4342-ab51-20eb70cdd9a2" (UID: "626c13b1-be66-4342-ab51-20eb70cdd9a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:25.212974 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.212265    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/626c13b1-be66-4342-ab51-20eb70cdd9a2-istiod-ca-cert" pod "626c13b1-be66-4342-ab51-20eb70cdd9a2" (UID: "626c13b1-be66-4342-ab51-20eb70cdd9a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:25.212974 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.212518    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-workload-certs" pod "08b34bfd-c225-46b7-adcc-2999244af328" (UID: "08b34bfd-c225-46b7-adcc-2999244af328"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:25.216096 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.215234    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/626c13b1-be66-4342-ab51-20eb70cdd9a2-kube-api-access-w4vzj" pod "626c13b1-be66-4342-ab51-20eb70cdd9a2" (UID: "626c13b1-be66-4342-ab51-20eb70cdd9a2"). InnerVolumeSpecName "kube-api-access-w4vzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:25.216096 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.215528    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b34bfd-c225-46b7-adcc-2999244af328-istiod-ca-cert" pod "08b34bfd-c225-46b7-adcc-2999244af328" (UID: "08b34bfd-c225-46b7-adcc-2999244af328"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:25.216096 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.215759    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-credential-socket" pod "626c13b1-be66-4342-ab51-20eb70cdd9a2" (UID: "626c13b1-be66-4342-ab51-20eb70cdd9a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:25.216096 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.216041    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-workload-certs" pod "626c13b1-be66-4342-ab51-20eb70cdd9a2" (UID: "626c13b1-be66-4342-ab51-20eb70cdd9a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:25.217170 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.217135    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-token" pod "626c13b1-be66-4342-ab51-20eb70cdd9a2" (UID: "626c13b1-be66-4342-ab51-20eb70cdd9a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:25.217436 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.217394    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08b34bfd-c225-46b7-adcc-2999244af328-istio-podinfo" pod "08b34bfd-c225-46b7-adcc-2999244af328" (UID: "08b34bfd-c225-46b7-adcc-2999244af328"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:27:25.220178 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.217672    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-podinfo" pod "626c13b1-be66-4342-ab51-20eb70cdd9a2" (UID: "626c13b1-be66-4342-ab51-20eb70cdd9a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:27:25.220178 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.219087    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b34bfd-c225-46b7-adcc-2999244af328-istio-token" pod "08b34bfd-c225-46b7-adcc-2999244af328" (UID: "08b34bfd-c225-46b7-adcc-2999244af328"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:25.222183 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.222106    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-istio-envoy" pod "08b34bfd-c225-46b7-adcc-2999244af328" (UID: "08b34bfd-c225-46b7-adcc-2999244af328"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:25.236939 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.232271    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-envoy" pod "626c13b1-be66-4342-ab51-20eb70cdd9a2" (UID: "626c13b1-be66-4342-ab51-20eb70cdd9a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:25.301868 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.301841    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff"]
Nov 03 00:27:25.306450 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.306416    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd7242d-9d05-487a-8d54-78667f412229-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67\" (UID: \"dbd7242d-9d05-487a-8d54-78667f412229\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67"
Nov 03 00:27:25.306450 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.306435    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff"
Nov 03 00:27:25.306592 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.306545    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkczf\" (UniqueName: \"kubernetes.io/projected/dbd7242d-9d05-487a-8d54-78667f412229-kube-api-access-vkczf\") pod \"payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67\" (UID: \"dbd7242d-9d05-487a-8d54-78667f412229\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67"
Nov 03 00:27:25.306691 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.306676    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.306727 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.306695    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b34bfd-c225-46b7-adcc-2999244af328-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.306727 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.306707    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.306727 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.306719    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.306824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.306731    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08b34bfd-c225-46b7-adcc-2999244af328-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.306824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.306742    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.306824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.306766    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.306824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.306778    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.306824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.306790    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.306824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.306803    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/626c13b1-be66-4342-ab51-20eb70cdd9a2-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.306824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.306815    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgdr8\" (UniqueName: \"kubernetes.io/projected/08b34bfd-c225-46b7-adcc-2999244af328-kube-api-access-zgdr8\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.307080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.306827    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4vzj\" (UniqueName: \"kubernetes.io/projected/626c13b1-be66-4342-ab51-20eb70cdd9a2-kube-api-access-w4vzj\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.307080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.306840    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b34bfd-c225-46b7-adcc-2999244af328-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.307080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.306852    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b34bfd-c225-46b7-adcc-2999244af328-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.307080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.306863    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/626c13b1-be66-4342-ab51-20eb70cdd9a2-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:25.330022 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.329984    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff"]
Nov 03 00:27:25.408083 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.408007    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkczf\" (UniqueName: \"kubernetes.io/projected/dbd7242d-9d05-487a-8d54-78667f412229-kube-api-access-vkczf\") pod \"payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67\" (UID: \"dbd7242d-9d05-487a-8d54-78667f412229\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67"
Nov 03 00:27:25.408247 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.408119    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff\" (UID: \"7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff"
Nov 03 00:27:25.408247 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.408190    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqkk5\" (UniqueName: \"kubernetes.io/projected/7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234-kube-api-access-tqkk5\") pod \"payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff\" (UID: \"7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff"
Nov 03 00:27:25.408391 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.408296    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd7242d-9d05-487a-8d54-78667f412229-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67\" (UID: \"dbd7242d-9d05-487a-8d54-78667f412229\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67"
Nov 03 00:27:25.409029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.409007    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd7242d-9d05-487a-8d54-78667f412229-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67\" (UID: \"dbd7242d-9d05-487a-8d54-78667f412229\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67"
Nov 03 00:27:25.418262 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.418234    5012 generic.go:356] "Generic (PLEG): container finished" podID="3684a36f-b227-463d-af7f-e8472ce6a45a" containerID="b537504792fcdd667e9c372fcf794ac237e3c79f6e05bdb2ef2023df5ae32602" exitCode=0
Nov 03 00:27:25.418398 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.418313    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"
Nov 03 00:27:25.418488 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.418409    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57" event={"ID":"3684a36f-b227-463d-af7f-e8472ce6a45a","Type":"ContainerDied","Data":"b537504792fcdd667e9c372fcf794ac237e3c79f6e05bdb2ef2023df5ae32602"}
Nov 03 00:27:25.418488 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.418470    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57" event={"ID":"3684a36f-b227-463d-af7f-e8472ce6a45a","Type":"ContainerDied","Data":"a42111fe6e8415f850d6f2592326d24deedd0aefc3355c0c82acf9481296d799"}
Nov 03 00:27:25.418602 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.418501    5012 scope.go:122] "RemoveContainer" containerID="b537504792fcdd667e9c372fcf794ac237e3c79f6e05bdb2ef2023df5ae32602"
Nov 03 00:27:25.420682 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.420650    5012 generic.go:356] "Generic (PLEG): container finished" podID="08b34bfd-c225-46b7-adcc-2999244af328" containerID="7115df497578842fb80ad63c681908cde635043019ae9a404fe591159d8981f4" exitCode=0
Nov 03 00:27:25.421065 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.421026    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x" event={"ID":"08b34bfd-c225-46b7-adcc-2999244af328","Type":"ContainerDied","Data":"7115df497578842fb80ad63c681908cde635043019ae9a404fe591159d8981f4"}
Nov 03 00:27:25.421395 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.421358    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x" event={"ID":"08b34bfd-c225-46b7-adcc-2999244af328","Type":"ContainerDied","Data":"f0f089238e5241219cf52065675efe5fe7d50cb4b28fecba5c1d1596f8e8325f"}
Nov 03 00:27:25.421395 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.421044    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"
Nov 03 00:27:25.422970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.422948    5012 generic.go:356] "Generic (PLEG): container finished" podID="626c13b1-be66-4342-ab51-20eb70cdd9a2" containerID="298b4d5fe81e474fbdd860e7fadf671b9979e3b1112c680e1f38f8e7c35dfe96" exitCode=0
Nov 03 00:27:25.423074 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.422971    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8" event={"ID":"626c13b1-be66-4342-ab51-20eb70cdd9a2","Type":"ContainerDied","Data":"298b4d5fe81e474fbdd860e7fadf671b9979e3b1112c680e1f38f8e7c35dfe96"}
Nov 03 00:27:25.423074 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.423009    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"
Nov 03 00:27:25.423074 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.423023    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8" event={"ID":"626c13b1-be66-4342-ab51-20eb70cdd9a2","Type":"ContainerDied","Data":"ecfe4818676d949006b4c4283acaccc5f992115549fea5ff2e53b1efbcce5624"}
Nov 03 00:27:25.428349 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.428323    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkczf\" (UniqueName: \"kubernetes.io/projected/dbd7242d-9d05-487a-8d54-78667f412229-kube-api-access-vkczf\") pod \"payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67\" (UID: \"dbd7242d-9d05-487a-8d54-78667f412229\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67"
Nov 03 00:27:25.431902 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.431872    5012 scope.go:122] "RemoveContainer" containerID="b537504792fcdd667e9c372fcf794ac237e3c79f6e05bdb2ef2023df5ae32602"
Nov 03 00:27:25.432235 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:27:25.432211    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b537504792fcdd667e9c372fcf794ac237e3c79f6e05bdb2ef2023df5ae32602\": container with ID starting with b537504792fcdd667e9c372fcf794ac237e3c79f6e05bdb2ef2023df5ae32602 not found: ID does not exist" containerID="b537504792fcdd667e9c372fcf794ac237e3c79f6e05bdb2ef2023df5ae32602"
Nov 03 00:27:25.432334 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.432243    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b537504792fcdd667e9c372fcf794ac237e3c79f6e05bdb2ef2023df5ae32602"} err="failed to get container status \"b537504792fcdd667e9c372fcf794ac237e3c79f6e05bdb2ef2023df5ae32602\": rpc error: code = NotFound desc = could not find container \"b537504792fcdd667e9c372fcf794ac237e3c79f6e05bdb2ef2023df5ae32602\": container with ID starting with b537504792fcdd667e9c372fcf794ac237e3c79f6e05bdb2ef2023df5ae32602 not found: ID does not exist"
Nov 03 00:27:25.432334 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.432262    5012 scope.go:122] "RemoveContainer" containerID="7115df497578842fb80ad63c681908cde635043019ae9a404fe591159d8981f4"
Nov 03 00:27:25.442673 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.442657    5012 scope.go:122] "RemoveContainer" containerID="7115df497578842fb80ad63c681908cde635043019ae9a404fe591159d8981f4"
Nov 03 00:27:25.442997 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:27:25.442979    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7115df497578842fb80ad63c681908cde635043019ae9a404fe591159d8981f4\": container with ID starting with 7115df497578842fb80ad63c681908cde635043019ae9a404fe591159d8981f4 not found: ID does not exist" containerID="7115df497578842fb80ad63c681908cde635043019ae9a404fe591159d8981f4"
Nov 03 00:27:25.443061 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.443002    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7115df497578842fb80ad63c681908cde635043019ae9a404fe591159d8981f4"} err="failed to get container status \"7115df497578842fb80ad63c681908cde635043019ae9a404fe591159d8981f4\": rpc error: code = NotFound desc = could not find container \"7115df497578842fb80ad63c681908cde635043019ae9a404fe591159d8981f4\": container with ID starting with 7115df497578842fb80ad63c681908cde635043019ae9a404fe591159d8981f4 not found: ID does not exist"
Nov 03 00:27:25.443061 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.443018    5012 scope.go:122] "RemoveContainer" containerID="298b4d5fe81e474fbdd860e7fadf671b9979e3b1112c680e1f38f8e7c35dfe96"
Nov 03 00:27:25.456312 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.456289    5012 scope.go:122] "RemoveContainer" containerID="298b4d5fe81e474fbdd860e7fadf671b9979e3b1112c680e1f38f8e7c35dfe96"
Nov 03 00:27:25.456597 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:27:25.456577    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"298b4d5fe81e474fbdd860e7fadf671b9979e3b1112c680e1f38f8e7c35dfe96\": container with ID starting with 298b4d5fe81e474fbdd860e7fadf671b9979e3b1112c680e1f38f8e7c35dfe96 not found: ID does not exist" containerID="298b4d5fe81e474fbdd860e7fadf671b9979e3b1112c680e1f38f8e7c35dfe96"
Nov 03 00:27:25.456671 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.456602    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"298b4d5fe81e474fbdd860e7fadf671b9979e3b1112c680e1f38f8e7c35dfe96"} err="failed to get container status \"298b4d5fe81e474fbdd860e7fadf671b9979e3b1112c680e1f38f8e7c35dfe96\": rpc error: code = NotFound desc = could not find container \"298b4d5fe81e474fbdd860e7fadf671b9979e3b1112c680e1f38f8e7c35dfe96\": container with ID starting with 298b4d5fe81e474fbdd860e7fadf671b9979e3b1112c680e1f38f8e7c35dfe96 not found: ID does not exist"
Nov 03 00:27:25.468795 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.468768    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"]
Nov 03 00:27:25.473256 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.473235    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-5rk6x"]
Nov 03 00:27:25.505165 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.505140    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"]
Nov 03 00:27:25.510305 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.510281    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6gwd8"]
Nov 03 00:27:25.511221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.511193    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff\" (UID: \"7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff"
Nov 03 00:27:25.511327 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.511303    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqkk5\" (UniqueName: \"kubernetes.io/projected/7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234-kube-api-access-tqkk5\") pod \"payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff\" (UID: \"7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff"
Nov 03 00:27:25.511923 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.511893    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff\" (UID: \"7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff"
Nov 03 00:27:25.539960 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.531236    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqkk5\" (UniqueName: \"kubernetes.io/projected/7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234-kube-api-access-tqkk5\") pod \"payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff\" (UID: \"7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff"
Nov 03 00:27:25.546747 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.546707    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"]
Nov 03 00:27:25.550027 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.550001    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67"
Nov 03 00:27:25.552416 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.552369    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rhx57"]
Nov 03 00:27:25.620194 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.619971    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff"
Nov 03 00:27:25.862664 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.862631    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67"]
Nov 03 00:27:26.003014 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:25.992698    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff"]
Nov 03 00:27:26.003014 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:27:25.993391    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef4f6ca_5dc0_4168_bf5e_2a9a3efd3234.slice/crio-4876f429898ccebabdbed39e44b4f384df87c5c7aad3464816e77bbddb761e03 WatchSource:0}: Error finding container 4876f429898ccebabdbed39e44b4f384df87c5c7aad3464816e77bbddb761e03: Status 404 returned error can't find the container with id 4876f429898ccebabdbed39e44b4f384df87c5c7aad3464816e77bbddb761e03
Nov 03 00:27:26.437309 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:26.437275    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff" event={"ID":"7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234","Type":"ContainerStarted","Data":"90b218e6ae0dfc758f6167290acb92134c0d09bf127927cd39aba0b54c448baf"}
Nov 03 00:27:26.437496 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:26.437337    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff"
Nov 03 00:27:26.437496 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:26.437360    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff" event={"ID":"7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234","Type":"ContainerStarted","Data":"4876f429898ccebabdbed39e44b4f384df87c5c7aad3464816e77bbddb761e03"}
Nov 03 00:27:26.439215 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:26.439182    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67" event={"ID":"dbd7242d-9d05-487a-8d54-78667f412229","Type":"ContainerStarted","Data":"75daaaa76d5c7ec2a7d05eb70c4143df5c1d61f41a08ff51c2cbd3f37a7f2bb7"}
Nov 03 00:27:26.439364 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:26.439239    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67" event={"ID":"dbd7242d-9d05-487a-8d54-78667f412229","Type":"ContainerStarted","Data":"89aacef159632237fb1cd1dc6205359025fdcc2a288e0dd9444d65d31c32643f"}
Nov 03 00:27:26.439364 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:26.439282    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67"
Nov 03 00:27:26.440886 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:26.440864    5012 generic.go:356] "Generic (PLEG): container finished" podID="94d99e83-8a4e-4d07-a350-571f916a764f" containerID="9dd6325444308fcad6adba4fbf8610961bdaec6e4291a280cbec5f1b9b731ada" exitCode=0
Nov 03 00:27:26.441039 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:26.440947    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76" event={"ID":"94d99e83-8a4e-4d07-a350-571f916a764f","Type":"ContainerDied","Data":"9dd6325444308fcad6adba4fbf8610961bdaec6e4291a280cbec5f1b9b731ada"}
Nov 03 00:27:26.441272 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:26.441255    5012 scope.go:122] "RemoveContainer" containerID="9dd6325444308fcad6adba4fbf8610961bdaec6e4291a280cbec5f1b9b731ada"
Nov 03 00:27:26.442772 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:26.442743    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:27:26.442858 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:26.442809    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:27:26.442858 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:26.442841    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:27:26.476135 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:26.476087    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff" podStartSLOduration=1.476074502 podStartE2EDuration="1.476074502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:27:26.472117977 +0000 UTC m=+1329.351702021" watchObservedRunningTime="2026-11-03 00:27:26.476074502 +0000 UTC m=+1329.355658544"
Nov 03 00:27:26.568570 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:26.568509    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67" podStartSLOduration=1.568490427 podStartE2EDuration="1.568490427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:27:26.5572398 +0000 UTC m=+1329.436823844" watchObservedRunningTime="2026-11-03 00:27:26.568490427 +0000 UTC m=+1329.448074472"
Nov 03 00:27:27.245208 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:27.245177    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08b34bfd-c225-46b7-adcc-2999244af328" path="/var/lib/kubelet/pods/08b34bfd-c225-46b7-adcc-2999244af328/volumes"
Nov 03 00:27:27.245821 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:27.245799    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3684a36f-b227-463d-af7f-e8472ce6a45a" path="/var/lib/kubelet/pods/3684a36f-b227-463d-af7f-e8472ce6a45a/volumes"
Nov 03 00:27:27.246300 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:27.246284    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="626c13b1-be66-4342-ab51-20eb70cdd9a2" path="/var/lib/kubelet/pods/626c13b1-be66-4342-ab51-20eb70cdd9a2/volumes"
Nov 03 00:27:27.452628 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:27.452589    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76" event={"ID":"94d99e83-8a4e-4d07-a350-571f916a764f","Type":"ContainerStarted","Data":"cadd1a11dada131dcdd96a3f0cd33734ebc9deef6ab5aaae3c8b22bbaf1f874b"}
Nov 03 00:27:28.421929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:28.421884    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:27:28.422500 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:28.421958    5012 patch_prober.go:29] interesting pod/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 03 00:27:28.422500 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:28.422013    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76" podUID="94d99e83-8a4e-4d07-a350-571f916a764f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 03 00:27:28.422500 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:28.422089    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f"
Nov 03 00:27:29.422424 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:29.422388    5012 patch_prober.go:29] interesting pod/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 03 00:27:29.422828 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:29.422452    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76" podUID="94d99e83-8a4e-4d07-a350-571f916a764f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 03 00:27:30.422799 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:30.422761    5012 patch_prober.go:29] interesting pod/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 03 00:27:30.423262 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:30.422823    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76" podUID="94d99e83-8a4e-4d07-a350-571f916a764f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 03 00:27:31.422412 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:31.422372    5012 patch_prober.go:29] interesting pod/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 03 00:27:31.422647 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:31.422436    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76" podUID="94d99e83-8a4e-4d07-a350-571f916a764f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 03 00:27:32.422542 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:32.422498    5012 patch_prober.go:29] interesting pod/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 03 00:27:32.422965 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:32.422562    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76" podUID="94d99e83-8a4e-4d07-a350-571f916a764f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 03 00:27:33.093681 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:33.093645    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"]
Nov 03 00:27:33.093940 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:33.093895    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4" podUID="b4accff6-4a81-4c76-9b46-417203ec2921" containerName="istio-proxy" containerID="cri-o://ce4b653514fab5edac08590d4c7784ccd4eb655cfba7a723e9da3eb8e53d8dae" gracePeriod=30
Nov 03 00:27:33.427650 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:33.427614    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:27:33.428058 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:33.428017    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:27:33.428956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:33.428900    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:27:33.474291 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:33.474258    5012 patch_prober.go:29] interesting pod/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:27:33.474452 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:33.474325    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4" podUID="b4accff6-4a81-4c76-9b46-417203ec2921" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:27:35.107485 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:35.107451    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"]
Nov 03 00:27:35.108001 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:35.107686    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps" podUID="d88041b1-e790-4f59-aee5-a053132ef8a4" containerName="istio-proxy" containerID="cri-o://3aa7adb14f74bf332210f1ab357dc88dee2e86a2243a3ebd7a35303281f4de95" gracePeriod=30
Nov 03 00:27:37.363228 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:27:37.363193    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:27:37.455284 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:37.455259    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff"
Nov 03 00:27:37.455430 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:37.455414    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67"
Nov 03 00:27:38.206182 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.206149    5012 patch_prober.go:29] interesting pod/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:27:38.206305 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.206212    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps" podUID="d88041b1-e790-4f59-aee5-a053132ef8a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:27:38.331896 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.331867    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj"]
Nov 03 00:27:38.360550 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.360523    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj"]
Nov 03 00:27:38.370197 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.370133    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:27:38.430298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.430263    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4accff6-4a81-4c76-9b46-417203ec2921-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4accff6-4a81-4c76-9b46-417203ec2921-istio-podinfo\") pod \"b4accff6-4a81-4c76-9b46-417203ec2921\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") "
Nov 03 00:27:38.430457 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.430340    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-workload-socket\") pod \"b4accff6-4a81-4c76-9b46-417203ec2921\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") "
Nov 03 00:27:38.430457 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.430407    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-istio-data\") pod \"b4accff6-4a81-4c76-9b46-417203ec2921\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") "
Nov 03 00:27:38.430593 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.430464    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4accff6-4a81-4c76-9b46-417203ec2921-kube-api-access-h5959\" (UniqueName: \"kubernetes.io/projected/b4accff6-4a81-4c76-9b46-417203ec2921-kube-api-access-h5959\") pod \"b4accff6-4a81-4c76-9b46-417203ec2921\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") "
Nov 03 00:27:38.430593 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.430512    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4accff6-4a81-4c76-9b46-417203ec2921-istio-token\" (UniqueName: \"kubernetes.io/projected/b4accff6-4a81-4c76-9b46-417203ec2921-istio-token\") pod \"b4accff6-4a81-4c76-9b46-417203ec2921\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") "
Nov 03 00:27:38.430694 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.430596    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-workload-certs\") pod \"b4accff6-4a81-4c76-9b46-417203ec2921\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") "
Nov 03 00:27:38.430694 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.430671    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-workload-socket" pod "b4accff6-4a81-4c76-9b46-417203ec2921" (UID: "b4accff6-4a81-4c76-9b46-417203ec2921"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:38.430694 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.430689    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4accff6-4a81-4c76-9b46-417203ec2921-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4accff6-4a81-4c76-9b46-417203ec2921-istiod-ca-cert\") pod \"b4accff6-4a81-4c76-9b46-417203ec2921\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") "
Nov 03 00:27:38.430864 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.430746    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-credential-socket\") pod \"b4accff6-4a81-4c76-9b46-417203ec2921\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") "
Nov 03 00:27:38.430864 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.430819    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-istio-envoy\") pod \"b4accff6-4a81-4c76-9b46-417203ec2921\" (UID: \"b4accff6-4a81-4c76-9b46-417203ec2921\") "
Nov 03 00:27:38.431216 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.431195    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-workload-certs" pod "b4accff6-4a81-4c76-9b46-417203ec2921" (UID: "b4accff6-4a81-4c76-9b46-417203ec2921"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:38.431719 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.431432    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:38.431719 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.431458    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:38.431719 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.431458    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-credential-socket" pod "b4accff6-4a81-4c76-9b46-417203ec2921" (UID: "b4accff6-4a81-4c76-9b46-417203ec2921"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:38.431719 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.431611    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4accff6-4a81-4c76-9b46-417203ec2921-istiod-ca-cert" pod "b4accff6-4a81-4c76-9b46-417203ec2921" (UID: "b4accff6-4a81-4c76-9b46-417203ec2921"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:38.432266 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.432244    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-istio-data" pod "b4accff6-4a81-4c76-9b46-417203ec2921" (UID: "b4accff6-4a81-4c76-9b46-417203ec2921"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:38.434456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.434286    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-istio-envoy" pod "b4accff6-4a81-4c76-9b46-417203ec2921" (UID: "b4accff6-4a81-4c76-9b46-417203ec2921"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:38.434543 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.434510    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4accff6-4a81-4c76-9b46-417203ec2921-kube-api-access-h5959" pod "b4accff6-4a81-4c76-9b46-417203ec2921" (UID: "b4accff6-4a81-4c76-9b46-417203ec2921"). InnerVolumeSpecName "kube-api-access-h5959". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:38.434631 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.434609    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4accff6-4a81-4c76-9b46-417203ec2921-istio-podinfo" pod "b4accff6-4a81-4c76-9b46-417203ec2921" (UID: "b4accff6-4a81-4c76-9b46-417203ec2921"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:27:38.434783 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.434759    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4accff6-4a81-4c76-9b46-417203ec2921-istio-token" pod "b4accff6-4a81-4c76-9b46-417203ec2921" (UID: "b4accff6-4a81-4c76-9b46-417203ec2921"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:38.533187 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.533157    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:38.533187 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.533184    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5959\" (UniqueName: \"kubernetes.io/projected/b4accff6-4a81-4c76-9b46-417203ec2921-kube-api-access-h5959\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:38.533187 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.533194    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4accff6-4a81-4c76-9b46-417203ec2921-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:38.533402 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.533202    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4accff6-4a81-4c76-9b46-417203ec2921-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:38.533402 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.533211    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:38.533402 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.533218    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4accff6-4a81-4c76-9b46-417203ec2921-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:38.533402 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.533227    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4accff6-4a81-4c76-9b46-417203ec2921-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:38.591085 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.591050    5012 generic.go:356] "Generic (PLEG): container finished" podID="b4accff6-4a81-4c76-9b46-417203ec2921" containerID="ce4b653514fab5edac08590d4c7784ccd4eb655cfba7a723e9da3eb8e53d8dae" exitCode=0
Nov 03 00:27:38.591242 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.591099    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"
Nov 03 00:27:38.591242 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.591137    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4" event={"ID":"b4accff6-4a81-4c76-9b46-417203ec2921","Type":"ContainerDied","Data":"ce4b653514fab5edac08590d4c7784ccd4eb655cfba7a723e9da3eb8e53d8dae"}
Nov 03 00:27:38.591242 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.591198    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4" event={"ID":"b4accff6-4a81-4c76-9b46-417203ec2921","Type":"ContainerDied","Data":"b51d190ac0d70eb9392a64886674609f49a0c6c634cac139fce35a085c1ac8d6"}
Nov 03 00:27:38.591242 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.591222    5012 scope.go:122] "RemoveContainer" containerID="ce4b653514fab5edac08590d4c7784ccd4eb655cfba7a723e9da3eb8e53d8dae"
Nov 03 00:27:38.603855 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.603835    5012 scope.go:122] "RemoveContainer" containerID="a24155cc71d930475106728162884b1d7f972a0eea49a6ad023877410eaac831"
Nov 03 00:27:38.614961 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.614895    5012 scope.go:122] "RemoveContainer" containerID="ce4b653514fab5edac08590d4c7784ccd4eb655cfba7a723e9da3eb8e53d8dae"
Nov 03 00:27:38.615251 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:27:38.615229    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce4b653514fab5edac08590d4c7784ccd4eb655cfba7a723e9da3eb8e53d8dae\": container with ID starting with ce4b653514fab5edac08590d4c7784ccd4eb655cfba7a723e9da3eb8e53d8dae not found: ID does not exist" containerID="ce4b653514fab5edac08590d4c7784ccd4eb655cfba7a723e9da3eb8e53d8dae"
Nov 03 00:27:38.615328 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.615260    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce4b653514fab5edac08590d4c7784ccd4eb655cfba7a723e9da3eb8e53d8dae"} err="failed to get container status \"ce4b653514fab5edac08590d4c7784ccd4eb655cfba7a723e9da3eb8e53d8dae\": rpc error: code = NotFound desc = could not find container \"ce4b653514fab5edac08590d4c7784ccd4eb655cfba7a723e9da3eb8e53d8dae\": container with ID starting with ce4b653514fab5edac08590d4c7784ccd4eb655cfba7a723e9da3eb8e53d8dae not found: ID does not exist"
Nov 03 00:27:38.615328 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.615277    5012 scope.go:122] "RemoveContainer" containerID="a24155cc71d930475106728162884b1d7f972a0eea49a6ad023877410eaac831"
Nov 03 00:27:38.615531 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:27:38.615510    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a24155cc71d930475106728162884b1d7f972a0eea49a6ad023877410eaac831\": container with ID starting with a24155cc71d930475106728162884b1d7f972a0eea49a6ad023877410eaac831 not found: ID does not exist" containerID="a24155cc71d930475106728162884b1d7f972a0eea49a6ad023877410eaac831"
Nov 03 00:27:38.615574 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.615539    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24155cc71d930475106728162884b1d7f972a0eea49a6ad023877410eaac831"} err="failed to get container status \"a24155cc71d930475106728162884b1d7f972a0eea49a6ad023877410eaac831\": rpc error: code = NotFound desc = could not find container \"a24155cc71d930475106728162884b1d7f972a0eea49a6ad023877410eaac831\": container with ID starting with a24155cc71d930475106728162884b1d7f972a0eea49a6ad023877410eaac831 not found: ID does not exist"
Nov 03 00:27:38.639459 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.639387    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"]
Nov 03 00:27:38.646871 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.646842    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-493ec8-openshift-default-76b589f5c7-wbbs4"]
Nov 03 00:27:38.690610 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.690580    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c8e7371-7s2q4"]
Nov 03 00:27:38.691419 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.691394    5012 cpu_manager.go:414] "containerMap: removing container" podUID="b4accff6-4a81-4c76-9b46-417203ec2921" containerName="istio-proxy"
Nov 03 00:27:38.691419 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.691414    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4accff6-4a81-4c76-9b46-417203ec2921" containerName="istio-proxy"
Nov 03 00:27:38.691572 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.691446    5012 cpu_manager.go:414] "containerMap: removing container" podUID="b4accff6-4a81-4c76-9b46-417203ec2921" containerName="istio-proxy"
Nov 03 00:27:38.691572 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.691457    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4accff6-4a81-4c76-9b46-417203ec2921" containerName="istio-proxy"
Nov 03 00:27:38.691679 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.691604    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4accff6-4a81-4c76-9b46-417203ec2921" containerName="istio-proxy"
Nov 03 00:27:38.691679 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.691618    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4accff6-4a81-4c76-9b46-417203ec2921" containerName="istio-proxy"
Nov 03 00:27:38.704980 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.696820    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8e7371-7s2q4"
Nov 03 00:27:38.726970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.724762    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:27:38.726970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.724889    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:27:38.813175 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.812884    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c8e7371-7s2q4"]
Nov 03 00:27:38.859328 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.859291    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4696ff14-ca42-478f-b026-6976d577cd88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c8e7371-7s2q4\" (UID: \"4696ff14-ca42-478f-b026-6976d577cd88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8e7371-7s2q4"
Nov 03 00:27:38.859541 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.859461    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4696ff14-ca42-478f-b026-6976d577cd88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c8e7371-7s2q4\" (UID: \"4696ff14-ca42-478f-b026-6976d577cd88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8e7371-7s2q4"
Nov 03 00:27:38.859541 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.859490    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk6h5\" (UniqueName: \"kubernetes.io/projected/4696ff14-ca42-478f-b026-6976d577cd88-kube-api-access-pk6h5\") pod \"maas-api-revoke-sub-8c8e7371-7s2q4\" (UID: \"4696ff14-ca42-478f-b026-6976d577cd88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8e7371-7s2q4"
Nov 03 00:27:38.961718 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.961556    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4696ff14-ca42-478f-b026-6976d577cd88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c8e7371-7s2q4\" (UID: \"4696ff14-ca42-478f-b026-6976d577cd88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8e7371-7s2q4"
Nov 03 00:27:38.962303 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.962270    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4696ff14-ca42-478f-b026-6976d577cd88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c8e7371-7s2q4\" (UID: \"4696ff14-ca42-478f-b026-6976d577cd88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8e7371-7s2q4"
Nov 03 00:27:38.962578 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.962537    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk6h5\" (UniqueName: \"kubernetes.io/projected/4696ff14-ca42-478f-b026-6976d577cd88-kube-api-access-pk6h5\") pod \"maas-api-revoke-sub-8c8e7371-7s2q4\" (UID: \"4696ff14-ca42-478f-b026-6976d577cd88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8e7371-7s2q4"
Nov 03 00:27:38.964821 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.964774    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4696ff14-ca42-478f-b026-6976d577cd88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c8e7371-7s2q4\" (UID: \"4696ff14-ca42-478f-b026-6976d577cd88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8e7371-7s2q4"
Nov 03 00:27:38.990973 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.982992    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4696ff14-ca42-478f-b026-6976d577cd88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c8e7371-7s2q4\" (UID: \"4696ff14-ca42-478f-b026-6976d577cd88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8e7371-7s2q4"
Nov 03 00:27:38.990973 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:38.983523    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk6h5\" (UniqueName: \"kubernetes.io/projected/4696ff14-ca42-478f-b026-6976d577cd88-kube-api-access-pk6h5\") pod \"maas-api-revoke-sub-8c8e7371-7s2q4\" (UID: \"4696ff14-ca42-478f-b026-6976d577cd88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8e7371-7s2q4"
Nov 03 00:27:39.115734 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:39.115689    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8e7371-7s2q4"
Nov 03 00:27:39.248574 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:39.248503    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4accff6-4a81-4c76-9b46-417203ec2921" path="/var/lib/kubelet/pods/b4accff6-4a81-4c76-9b46-417203ec2921/volumes"
Nov 03 00:27:39.478592 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:39.478558    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c8e7371-7s2q4"]
Nov 03 00:27:39.479055 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:27:39.479024    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4696ff14_ca42_478f_b026_6976d577cd88.slice/crio-dd6bcdecd28453d1fd8e9cbd1eee6b92568cda1c61690c2afa4445dcf8e46939 WatchSource:0}: Error finding container dd6bcdecd28453d1fd8e9cbd1eee6b92568cda1c61690c2afa4445dcf8e46939: Status 404 returned error can't find the container with id dd6bcdecd28453d1fd8e9cbd1eee6b92568cda1c61690c2afa4445dcf8e46939
Nov 03 00:27:39.489176 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:39.489148    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh"]
Nov 03 00:27:39.603625 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:39.603582    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8e7371-7s2q4" event={"ID":"4696ff14-ca42-478f-b026-6976d577cd88","Type":"ContainerStarted","Data":"dd6bcdecd28453d1fd8e9cbd1eee6b92568cda1c61690c2afa4445dcf8e46939"}
Nov 03 00:27:40.464115 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.464087    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:27:40.609347 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.609315    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-workload-socket\") pod \"d88041b1-e790-4f59-aee5-a053132ef8a4\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") "
Nov 03 00:27:40.609806 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.609360    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d88041b1-e790-4f59-aee5-a053132ef8a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d88041b1-e790-4f59-aee5-a053132ef8a4-istiod-ca-cert\") pod \"d88041b1-e790-4f59-aee5-a053132ef8a4\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") "
Nov 03 00:27:40.609806 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.609414    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-data\") pod \"d88041b1-e790-4f59-aee5-a053132ef8a4\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") "
Nov 03 00:27:40.609806 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.609466    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-credential-socket\") pod \"d88041b1-e790-4f59-aee5-a053132ef8a4\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") "
Nov 03 00:27:40.609806 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.609569    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-envoy\") pod \"d88041b1-e790-4f59-aee5-a053132ef8a4\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") "
Nov 03 00:27:40.609806 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.609614    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-workload-socket" pod "d88041b1-e790-4f59-aee5-a053132ef8a4" (UID: "d88041b1-e790-4f59-aee5-a053132ef8a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:40.609806 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.609648    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-podinfo\") pod \"d88041b1-e790-4f59-aee5-a053132ef8a4\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") "
Nov 03 00:27:40.609806 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.609699    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d88041b1-e790-4f59-aee5-a053132ef8a4-kube-api-access-2l8jg\" (UniqueName: \"kubernetes.io/projected/d88041b1-e790-4f59-aee5-a053132ef8a4-kube-api-access-2l8jg\") pod \"d88041b1-e790-4f59-aee5-a053132ef8a4\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") "
Nov 03 00:27:40.609806 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.609730    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-data" pod "d88041b1-e790-4f59-aee5-a053132ef8a4" (UID: "d88041b1-e790-4f59-aee5-a053132ef8a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:40.609806 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.609734    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-workload-certs\") pod \"d88041b1-e790-4f59-aee5-a053132ef8a4\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") "
Nov 03 00:27:40.610284 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.609817    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d88041b1-e790-4f59-aee5-a053132ef8a4-istiod-ca-cert" pod "d88041b1-e790-4f59-aee5-a053132ef8a4" (UID: "d88041b1-e790-4f59-aee5-a053132ef8a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:40.610284 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.609856    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-token\" (UniqueName: \"kubernetes.io/projected/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-token\") pod \"d88041b1-e790-4f59-aee5-a053132ef8a4\" (UID: \"d88041b1-e790-4f59-aee5-a053132ef8a4\") "
Nov 03 00:27:40.610284 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.609977    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-workload-certs" pod "d88041b1-e790-4f59-aee5-a053132ef8a4" (UID: "d88041b1-e790-4f59-aee5-a053132ef8a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:40.610284 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.610153    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-credential-socket" pod "d88041b1-e790-4f59-aee5-a053132ef8a4" (UID: "d88041b1-e790-4f59-aee5-a053132ef8a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:40.610680 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.610655    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:40.610774 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.610687    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d88041b1-e790-4f59-aee5-a053132ef8a4-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:40.610774 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.610702    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:40.610774 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.610718    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:40.610774 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.610732    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:40.611884 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.611861    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-envoy" pod "d88041b1-e790-4f59-aee5-a053132ef8a4" (UID: "d88041b1-e790-4f59-aee5-a053132ef8a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:40.612082 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.612054    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-token" pod "d88041b1-e790-4f59-aee5-a053132ef8a4" (UID: "d88041b1-e790-4f59-aee5-a053132ef8a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:40.612156 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.612078    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-podinfo" pod "d88041b1-e790-4f59-aee5-a053132ef8a4" (UID: "d88041b1-e790-4f59-aee5-a053132ef8a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:27:40.612292 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.612273    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d88041b1-e790-4f59-aee5-a053132ef8a4-kube-api-access-2l8jg" pod "d88041b1-e790-4f59-aee5-a053132ef8a4" (UID: "d88041b1-e790-4f59-aee5-a053132ef8a4"). InnerVolumeSpecName "kube-api-access-2l8jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:40.614204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.614175    5012 generic.go:356] "Generic (PLEG): container finished" podID="d88041b1-e790-4f59-aee5-a053132ef8a4" containerID="3aa7adb14f74bf332210f1ab357dc88dee2e86a2243a3ebd7a35303281f4de95" exitCode=0
Nov 03 00:27:40.614290 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.614230    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"
Nov 03 00:27:40.614290 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.614255    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps" event={"ID":"d88041b1-e790-4f59-aee5-a053132ef8a4","Type":"ContainerDied","Data":"3aa7adb14f74bf332210f1ab357dc88dee2e86a2243a3ebd7a35303281f4de95"}
Nov 03 00:27:40.614382 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.614317    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps" event={"ID":"d88041b1-e790-4f59-aee5-a053132ef8a4","Type":"ContainerDied","Data":"a4aa4dbd4b75abf695b322f3576a9bf1ce24042c37de74acc42465b856f0afbb"}
Nov 03 00:27:40.614382 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.614345    5012 scope.go:122] "RemoveContainer" containerID="3aa7adb14f74bf332210f1ab357dc88dee2e86a2243a3ebd7a35303281f4de95"
Nov 03 00:27:40.616386 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.616362    5012 generic.go:356] "Generic (PLEG): container finished" podID="4696ff14-ca42-478f-b026-6976d577cd88" containerID="6a7e5da61572eacda0356ce6dc7e2df213d86b4bbc8ff2fc5642950d37f7d780" exitCode=0
Nov 03 00:27:40.616478 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.616452    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8e7371-7s2q4" event={"ID":"4696ff14-ca42-478f-b026-6976d577cd88","Type":"ContainerDied","Data":"6a7e5da61572eacda0356ce6dc7e2df213d86b4bbc8ff2fc5642950d37f7d780"}
Nov 03 00:27:40.642602 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.642585    5012 scope.go:122] "RemoveContainer" containerID="50980e2e32c08dfcdcc3371c2c8831bc59936d37d3c65d0d703fa34607169e9c"
Nov 03 00:27:40.652147 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.652133    5012 scope.go:122] "RemoveContainer" containerID="3aa7adb14f74bf332210f1ab357dc88dee2e86a2243a3ebd7a35303281f4de95"
Nov 03 00:27:40.652418 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:27:40.652401    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aa7adb14f74bf332210f1ab357dc88dee2e86a2243a3ebd7a35303281f4de95\": container with ID starting with 3aa7adb14f74bf332210f1ab357dc88dee2e86a2243a3ebd7a35303281f4de95 not found: ID does not exist" containerID="3aa7adb14f74bf332210f1ab357dc88dee2e86a2243a3ebd7a35303281f4de95"
Nov 03 00:27:40.652463 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.652425    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa7adb14f74bf332210f1ab357dc88dee2e86a2243a3ebd7a35303281f4de95"} err="failed to get container status \"3aa7adb14f74bf332210f1ab357dc88dee2e86a2243a3ebd7a35303281f4de95\": rpc error: code = NotFound desc = could not find container \"3aa7adb14f74bf332210f1ab357dc88dee2e86a2243a3ebd7a35303281f4de95\": container with ID starting with 3aa7adb14f74bf332210f1ab357dc88dee2e86a2243a3ebd7a35303281f4de95 not found: ID does not exist"
Nov 03 00:27:40.652463 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.652442    5012 scope.go:122] "RemoveContainer" containerID="50980e2e32c08dfcdcc3371c2c8831bc59936d37d3c65d0d703fa34607169e9c"
Nov 03 00:27:40.652650 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:27:40.652633    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50980e2e32c08dfcdcc3371c2c8831bc59936d37d3c65d0d703fa34607169e9c\": container with ID starting with 50980e2e32c08dfcdcc3371c2c8831bc59936d37d3c65d0d703fa34607169e9c not found: ID does not exist" containerID="50980e2e32c08dfcdcc3371c2c8831bc59936d37d3c65d0d703fa34607169e9c"
Nov 03 00:27:40.652687 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.652657    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50980e2e32c08dfcdcc3371c2c8831bc59936d37d3c65d0d703fa34607169e9c"} err="failed to get container status \"50980e2e32c08dfcdcc3371c2c8831bc59936d37d3c65d0d703fa34607169e9c\": rpc error: code = NotFound desc = could not find container \"50980e2e32c08dfcdcc3371c2c8831bc59936d37d3c65d0d703fa34607169e9c\": container with ID starting with 50980e2e32c08dfcdcc3371c2c8831bc59936d37d3c65d0d703fa34607169e9c not found: ID does not exist"
Nov 03 00:27:40.697698 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.697634    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"]
Nov 03 00:27:40.701270 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.701250    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-cb05ae-openshift-default-7bc99bd8f9-6hcps"]
Nov 03 00:27:40.712314 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.712289    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:40.712314 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.712311    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:40.712428 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.712320    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l8jg\" (UniqueName: \"kubernetes.io/projected/d88041b1-e790-4f59-aee5-a053132ef8a4-kube-api-access-2l8jg\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:40.712428 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:40.712329    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d88041b1-e790-4f59-aee5-a053132ef8a4-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:41.245158 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.245125    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d88041b1-e790-4f59-aee5-a053132ef8a4" path="/var/lib/kubelet/pods/d88041b1-e790-4f59-aee5-a053132ef8a4/volumes"
Nov 03 00:27:41.579035 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.578957    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"]
Nov 03 00:27:41.579922 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.579874    5012 cpu_manager.go:414] "containerMap: removing container" podUID="d88041b1-e790-4f59-aee5-a053132ef8a4" containerName="istio-proxy"
Nov 03 00:27:41.579922 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.579899    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d88041b1-e790-4f59-aee5-a053132ef8a4" containerName="istio-proxy"
Nov 03 00:27:41.580120 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.579960    5012 cpu_manager.go:414] "containerMap: removing container" podUID="d88041b1-e790-4f59-aee5-a053132ef8a4" containerName="istio-proxy"
Nov 03 00:27:41.580120 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.579967    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d88041b1-e790-4f59-aee5-a053132ef8a4" containerName="istio-proxy"
Nov 03 00:27:41.580120 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.580105    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="d88041b1-e790-4f59-aee5-a053132ef8a4" containerName="istio-proxy"
Nov 03 00:27:41.580255 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.580124    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="d88041b1-e790-4f59-aee5-a053132ef8a4" containerName="istio-proxy"
Nov 03 00:27:41.585481 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.585462    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.609947 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.609595    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"]
Nov 03 00:27:41.611713 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.611691    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-e37c9d-openshift-default-dockercfg-4kk6c\""
Nov 03 00:27:41.793729 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.791969    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-istio-data\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.793729 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.792091    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba93722b-9169-4628-9f61-baaef18a68a7-istiod-ca-cert\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.798821 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.798770    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-workload-certs\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.798982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.798838    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kc22\" (UniqueName: \"kubernetes.io/projected/ba93722b-9169-4628-9f61-baaef18a68a7-kube-api-access-6kc22\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.814405 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.810112    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-workload-socket\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.814405 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.810222    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-istio-envoy\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.814405 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.810301    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba93722b-9169-4628-9f61-baaef18a68a7-istio-podinfo\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.814405 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.810338    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba93722b-9169-4628-9f61-baaef18a68a7-istio-token\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.814405 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.810369    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-credential-socket\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.911857 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.911819    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-istio-data\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.912003 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.911947    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba93722b-9169-4628-9f61-baaef18a68a7-istiod-ca-cert\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.912003 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.911983    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-workload-certs\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.912088 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.912027    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kc22\" (UniqueName: \"kubernetes.io/projected/ba93722b-9169-4628-9f61-baaef18a68a7-kube-api-access-6kc22\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.912136 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.912100    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-workload-socket\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.912202 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.912185    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-istio-envoy\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.912277 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.912263    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba93722b-9169-4628-9f61-baaef18a68a7-istio-podinfo\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.912325 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.912300    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba93722b-9169-4628-9f61-baaef18a68a7-istio-token\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.912358 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.912332    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-credential-socket\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.912827 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.912806    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-credential-socket\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.913306 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.913127    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-istio-data\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.913793 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.913765    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba93722b-9169-4628-9f61-baaef18a68a7-istiod-ca-cert\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.914077 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.914056    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-workload-certs\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.914622 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.914599    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-workload-socket\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.916945 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.916888    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-istio-envoy\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.918307 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.918287    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba93722b-9169-4628-9f61-baaef18a68a7-istio-podinfo\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.932401 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.932376    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba93722b-9169-4628-9f61-baaef18a68a7-istio-token\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.932524 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.932505    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kc22\" (UniqueName: \"kubernetes.io/projected/ba93722b-9169-4628-9f61-baaef18a68a7-kube-api-access-6kc22\") pod \"e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") " pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:41.959949 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:41.959929    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8e7371-7s2q4"
Nov 03 00:27:42.013052 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:42.013014    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4696ff14-ca42-478f-b026-6976d577cd88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4696ff14-ca42-478f-b026-6976d577cd88-maas-api-openshift-ca\") pod \"4696ff14-ca42-478f-b026-6976d577cd88\" (UID: \"4696ff14-ca42-478f-b026-6976d577cd88\") "
Nov 03 00:27:42.013232 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:42.013119    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4696ff14-ca42-478f-b026-6976d577cd88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4696ff14-ca42-478f-b026-6976d577cd88-maas-api-xks-ca\") pod \"4696ff14-ca42-478f-b026-6976d577cd88\" (UID: \"4696ff14-ca42-478f-b026-6976d577cd88\") "
Nov 03 00:27:42.013343 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:42.013318    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4696ff14-ca42-478f-b026-6976d577cd88-kube-api-access-pk6h5\" (UniqueName: \"kubernetes.io/projected/4696ff14-ca42-478f-b026-6976d577cd88-kube-api-access-pk6h5\") pod \"4696ff14-ca42-478f-b026-6976d577cd88\" (UID: \"4696ff14-ca42-478f-b026-6976d577cd88\") "
Nov 03 00:27:42.013460 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:42.013343    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4696ff14-ca42-478f-b026-6976d577cd88-maas-api-openshift-ca" pod "4696ff14-ca42-478f-b026-6976d577cd88" (UID: "4696ff14-ca42-478f-b026-6976d577cd88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:42.013977 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:42.013936    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4696ff14-ca42-478f-b026-6976d577cd88-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:42.015355 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:42.015325    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4696ff14-ca42-478f-b026-6976d577cd88-maas-api-xks-ca" pod "4696ff14-ca42-478f-b026-6976d577cd88" (UID: "4696ff14-ca42-478f-b026-6976d577cd88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:27:42.015857 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:42.015825    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4696ff14-ca42-478f-b026-6976d577cd88-kube-api-access-pk6h5" pod "4696ff14-ca42-478f-b026-6976d577cd88" (UID: "4696ff14-ca42-478f-b026-6976d577cd88"). InnerVolumeSpecName "kube-api-access-pk6h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:42.114811 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:42.114785    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk6h5\" (UniqueName: \"kubernetes.io/projected/4696ff14-ca42-478f-b026-6976d577cd88-kube-api-access-pk6h5\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:42.114811 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:42.114811    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4696ff14-ca42-478f-b026-6976d577cd88-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:42.203805 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:42.203742    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:42.515095 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:27:42.515064    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba93722b_9169_4628_9f61_baaef18a68a7.slice/crio-792161c75b5b7fb1bb5f40187cd36564f2794054028c33b19d4620d7560455c7 WatchSource:0}: Error finding container 792161c75b5b7fb1bb5f40187cd36564f2794054028c33b19d4620d7560455c7: Status 404 returned error can't find the container with id 792161c75b5b7fb1bb5f40187cd36564f2794054028c33b19d4620d7560455c7
Nov 03 00:27:42.515673 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:42.515645    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"]
Nov 03 00:27:42.518639 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:42.518596    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:27:42.518753 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:42.518705    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:27:42.518819 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:42.518775    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:27:42.639005 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:42.638977    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8e7371-7s2q4"
Nov 03 00:27:42.639465 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:42.639003    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8e7371-7s2q4" event={"ID":"4696ff14-ca42-478f-b026-6976d577cd88","Type":"ContainerDied","Data":"dd6bcdecd28453d1fd8e9cbd1eee6b92568cda1c61690c2afa4445dcf8e46939"}
Nov 03 00:27:42.639465 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:42.639087    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd6bcdecd28453d1fd8e9cbd1eee6b92568cda1c61690c2afa4445dcf8e46939"
Nov 03 00:27:42.641099 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:42.641073    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms" event={"ID":"ba93722b-9169-4628-9f61-baaef18a68a7","Type":"ContainerStarted","Data":"f4878d8508f1d06d42d201a2f1dcd80e113ab7824553830502fc5ce404ce1390"}
Nov 03 00:27:42.641232 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:42.641118    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms" event={"ID":"ba93722b-9169-4628-9f61-baaef18a68a7","Type":"ContainerStarted","Data":"792161c75b5b7fb1bb5f40187cd36564f2794054028c33b19d4620d7560455c7"}
Nov 03 00:27:42.682205 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:42.682158    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms" podStartSLOduration=1.682144807 podStartE2EDuration="1.682144807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:27:42.677897497 +0000 UTC m=+1345.557481542" watchObservedRunningTime="2026-11-03 00:27:42.682144807 +0000 UTC m=+1345.561728849"
Nov 03 00:27:43.203903 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:43.203869    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:43.209303 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:43.209279    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:43.332318 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:43.332286    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj" podUID="c9679570-209d-48af-9805-eed8398b4895" containerName="payload-processing" containerID="cri-o://93e454c3f86def1b02eb90e34b4922d5629b9ae508a6e8fc8beea02cd6b3fe27" gracePeriod=25
Nov 03 00:27:43.361449 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:43.361423    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj" podUID="75b8d3b6-0070-4826-b79d-618721119057" containerName="payload-pre-processing" containerID="cri-o://e39dd83175b6219a3192703190f404bababf8dde3e5de5eb2bbfc6ccab001cc7" gracePeriod=25
Nov 03 00:27:43.653268 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:43.652444    5012 generic.go:356] "Generic (PLEG): container finished" podID="c9679570-209d-48af-9805-eed8398b4895" containerID="93e454c3f86def1b02eb90e34b4922d5629b9ae508a6e8fc8beea02cd6b3fe27" exitCode=0
Nov 03 00:27:43.653268 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:43.652640    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj" event={"ID":"c9679570-209d-48af-9805-eed8398b4895","Type":"ContainerDied","Data":"93e454c3f86def1b02eb90e34b4922d5629b9ae508a6e8fc8beea02cd6b3fe27"}
Nov 03 00:27:43.654764 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:43.654740    5012 generic.go:356] "Generic (PLEG): container finished" podID="75b8d3b6-0070-4826-b79d-618721119057" containerID="e39dd83175b6219a3192703190f404bababf8dde3e5de5eb2bbfc6ccab001cc7" exitCode=0
Nov 03 00:27:43.654898 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:43.654815    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj" event={"ID":"75b8d3b6-0070-4826-b79d-618721119057","Type":"ContainerDied","Data":"e39dd83175b6219a3192703190f404bababf8dde3e5de5eb2bbfc6ccab001cc7"}
Nov 03 00:27:43.655237 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:43.655217    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:43.656331 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:43.656316    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:27:43.692687 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:43.692664    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj"
Nov 03 00:27:43.727020 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:43.726995    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj"
Nov 03 00:27:43.740805 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:43.737614    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9679570-209d-48af-9805-eed8398b4895-kube-api-access-xwnq5\" (UniqueName: \"kubernetes.io/projected/c9679570-209d-48af-9805-eed8398b4895-kube-api-access-xwnq5\") pod \"c9679570-209d-48af-9805-eed8398b4895\" (UID: \"c9679570-209d-48af-9805-eed8398b4895\") "
Nov 03 00:27:43.740805 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:43.737655    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9679570-209d-48af-9805-eed8398b4895-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9679570-209d-48af-9805-eed8398b4895-plugins-config-volume\") pod \"c9679570-209d-48af-9805-eed8398b4895\" (UID: \"c9679570-209d-48af-9805-eed8398b4895\") "
Nov 03 00:27:43.740805 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:43.739417    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9679570-209d-48af-9805-eed8398b4895-plugins-config-volume" pod "c9679570-209d-48af-9805-eed8398b4895" (UID: "c9679570-209d-48af-9805-eed8398b4895"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:43.740805 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:43.740275    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9679570-209d-48af-9805-eed8398b4895-kube-api-access-xwnq5" pod "c9679570-209d-48af-9805-eed8398b4895" (UID: "c9679570-209d-48af-9805-eed8398b4895"). InnerVolumeSpecName "kube-api-access-xwnq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:43.742279 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:43.741823    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwnq5\" (UniqueName: \"kubernetes.io/projected/c9679570-209d-48af-9805-eed8398b4895-kube-api-access-xwnq5\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:43.742279 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:43.741844    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9679570-209d-48af-9805-eed8398b4895-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:43.842277 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:43.842231    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b8d3b6-0070-4826-b79d-618721119057-kube-api-access-gz9x2\" (UniqueName: \"kubernetes.io/projected/75b8d3b6-0070-4826-b79d-618721119057-kube-api-access-gz9x2\") pod \"75b8d3b6-0070-4826-b79d-618721119057\" (UID: \"75b8d3b6-0070-4826-b79d-618721119057\") "
Nov 03 00:27:43.842772 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:43.842393    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75b8d3b6-0070-4826-b79d-618721119057-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75b8d3b6-0070-4826-b79d-618721119057-plugins-config-volume\") pod \"75b8d3b6-0070-4826-b79d-618721119057\" (UID: \"75b8d3b6-0070-4826-b79d-618721119057\") "
Nov 03 00:27:43.842772 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:43.842726    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b8d3b6-0070-4826-b79d-618721119057-plugins-config-volume" pod "75b8d3b6-0070-4826-b79d-618721119057" (UID: "75b8d3b6-0070-4826-b79d-618721119057"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:43.843063 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:43.843043    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75b8d3b6-0070-4826-b79d-618721119057-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:43.844199 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:43.844177    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b8d3b6-0070-4826-b79d-618721119057-kube-api-access-gz9x2" pod "75b8d3b6-0070-4826-b79d-618721119057" (UID: "75b8d3b6-0070-4826-b79d-618721119057"). InnerVolumeSpecName "kube-api-access-gz9x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:43.943926 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:43.943889    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz9x2\" (UniqueName: \"kubernetes.io/projected/75b8d3b6-0070-4826-b79d-618721119057-kube-api-access-gz9x2\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:44.489521 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:44.489485    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh" podUID="85ac030e-4b72-48bb-956c-90f21ca74de1" containerName="payload-processing" containerID="cri-o://0d5b1edd934576484744ec4005578ea1bc8bdee6b4caf11ab8f39d18ad0e2556" gracePeriod=25
Nov 03 00:27:44.665785 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:44.665407    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj"
Nov 03 00:27:44.665785 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:44.665437    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj" event={"ID":"c9679570-209d-48af-9805-eed8398b4895","Type":"ContainerDied","Data":"26c2353ea2f906b4f88001c77647006bffcc45dbba23e198744f16cdd8c33964"}
Nov 03 00:27:44.665785 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:44.665517    5012 scope.go:122] "RemoveContainer" containerID="93e454c3f86def1b02eb90e34b4922d5629b9ae508a6e8fc8beea02cd6b3fe27"
Nov 03 00:27:44.668355 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:44.668321    5012 generic.go:356] "Generic (PLEG): container finished" podID="85ac030e-4b72-48bb-956c-90f21ca74de1" containerID="0d5b1edd934576484744ec4005578ea1bc8bdee6b4caf11ab8f39d18ad0e2556" exitCode=0
Nov 03 00:27:44.668511 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:44.668492    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh" event={"ID":"85ac030e-4b72-48bb-956c-90f21ca74de1","Type":"ContainerDied","Data":"0d5b1edd934576484744ec4005578ea1bc8bdee6b4caf11ab8f39d18ad0e2556"}
Nov 03 00:27:44.671624 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:44.671601    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj"
Nov 03 00:27:44.672043 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:44.672022    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj" event={"ID":"75b8d3b6-0070-4826-b79d-618721119057","Type":"ContainerDied","Data":"32e82e40ab402fefdcc51b1ea8bafb89c368ae06a30e7b8376a52cd49a43dca4"}
Nov 03 00:27:44.698358 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:44.697208    5012 scope.go:122] "RemoveContainer" containerID="e39dd83175b6219a3192703190f404bababf8dde3e5de5eb2bbfc6ccab001cc7"
Nov 03 00:27:44.729619 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:44.729588    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj"]
Nov 03 00:27:44.738798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:44.738771    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dda6df-8697d79885-tl8xj"]
Nov 03 00:27:44.760287 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:44.760264    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj"]
Nov 03 00:27:44.764024 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:44.764001    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-85564fc6cf-llqtj"]
Nov 03 00:27:44.779559 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:44.779540    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh"
Nov 03 00:27:44.851546 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:44.851523    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85ac030e-4b72-48bb-956c-90f21ca74de1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85ac030e-4b72-48bb-956c-90f21ca74de1-plugins-config-volume\") pod \"85ac030e-4b72-48bb-956c-90f21ca74de1\" (UID: \"85ac030e-4b72-48bb-956c-90f21ca74de1\") "
Nov 03 00:27:44.851684 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:44.851576    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85ac030e-4b72-48bb-956c-90f21ca74de1-kube-api-access-9jtd5\" (UniqueName: \"kubernetes.io/projected/85ac030e-4b72-48bb-956c-90f21ca74de1-kube-api-access-9jtd5\") pod \"85ac030e-4b72-48bb-956c-90f21ca74de1\" (UID: \"85ac030e-4b72-48bb-956c-90f21ca74de1\") "
Nov 03 00:27:44.852011 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:44.851983    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ac030e-4b72-48bb-956c-90f21ca74de1-plugins-config-volume" pod "85ac030e-4b72-48bb-956c-90f21ca74de1" (UID: "85ac030e-4b72-48bb-956c-90f21ca74de1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:44.852135 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:44.852117    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85ac030e-4b72-48bb-956c-90f21ca74de1-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:44.853784 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:44.853762    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ac030e-4b72-48bb-956c-90f21ca74de1-kube-api-access-9jtd5" pod "85ac030e-4b72-48bb-956c-90f21ca74de1" (UID: "85ac030e-4b72-48bb-956c-90f21ca74de1"). InnerVolumeSpecName "kube-api-access-9jtd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:44.953056 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:44.953028    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jtd5\" (UniqueName: \"kubernetes.io/projected/85ac030e-4b72-48bb-956c-90f21ca74de1-kube-api-access-9jtd5\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:27:45.246274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:45.246238    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75b8d3b6-0070-4826-b79d-618721119057" path="/var/lib/kubelet/pods/75b8d3b6-0070-4826-b79d-618721119057/volumes"
Nov 03 00:27:45.246810 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:45.246786    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9679570-209d-48af-9805-eed8398b4895" path="/var/lib/kubelet/pods/c9679570-209d-48af-9805-eed8398b4895/volumes"
Nov 03 00:27:45.681853 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:45.681820    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh"
Nov 03 00:27:45.682361 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:45.681867    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh" event={"ID":"85ac030e-4b72-48bb-956c-90f21ca74de1","Type":"ContainerDied","Data":"56d925a58048398173ddbac849f78d311b7df290952a676bc55e6494470a4644"}
Nov 03 00:27:45.682361 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:45.681990    5012 scope.go:122] "RemoveContainer" containerID="0d5b1edd934576484744ec4005578ea1bc8bdee6b4caf11ab8f39d18ad0e2556"
Nov 03 00:27:45.733768 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:45.733735    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh"]
Nov 03 00:27:45.743472 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:45.743444    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d43409-7c59ffd85c-6f7bh"]
Nov 03 00:27:47.045307 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.045273    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9"]
Nov 03 00:27:47.045953 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.045936    5012 cpu_manager.go:414] "containerMap: removing container" podUID="c9679570-209d-48af-9805-eed8398b4895" containerName="payload-processing"
Nov 03 00:27:47.045953 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.045951    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9679570-209d-48af-9805-eed8398b4895" containerName="payload-processing"
Nov 03 00:27:47.046041 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.045985    5012 cpu_manager.go:414] "containerMap: removing container" podUID="75b8d3b6-0070-4826-b79d-618721119057" containerName="payload-pre-processing"
Nov 03 00:27:47.046041 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.045991    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b8d3b6-0070-4826-b79d-618721119057" containerName="payload-pre-processing"
Nov 03 00:27:47.046041 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.046007    5012 cpu_manager.go:414] "containerMap: removing container" podUID="85ac030e-4b72-48bb-956c-90f21ca74de1" containerName="payload-processing"
Nov 03 00:27:47.046041 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.046011    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85ac030e-4b72-48bb-956c-90f21ca74de1" containerName="payload-processing"
Nov 03 00:27:47.046041 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.046021    5012 cpu_manager.go:414] "containerMap: removing container" podUID="4696ff14-ca42-478f-b026-6976d577cd88" containerName="revoke-keys"
Nov 03 00:27:47.046041 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.046026    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4696ff14-ca42-478f-b026-6976d577cd88" containerName="revoke-keys"
Nov 03 00:27:47.046223 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.046112    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b8d3b6-0070-4826-b79d-618721119057" containerName="payload-pre-processing"
Nov 03 00:27:47.046223 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.046124    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="85ac030e-4b72-48bb-956c-90f21ca74de1" containerName="payload-processing"
Nov 03 00:27:47.046223 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.046130    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="4696ff14-ca42-478f-b026-6976d577cd88" containerName="revoke-keys"
Nov 03 00:27:47.046223 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.046139    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9679570-209d-48af-9805-eed8398b4895" containerName="payload-processing"
Nov 03 00:27:47.050854 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.050835    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9"
Nov 03 00:27:47.076306 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.076282    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9"]
Nov 03 00:27:47.081202 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.081166    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e37c9d\""
Nov 03 00:27:47.087268 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.087242    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59ds2\" (UniqueName: \"kubernetes.io/projected/911edb88-1e60-40b0-843e-7086beabbc9b-kube-api-access-59ds2\") pod \"payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9\" (UID: \"911edb88-1e60-40b0-843e-7086beabbc9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9"
Nov 03 00:27:47.087363 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.087291    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911edb88-1e60-40b0-843e-7086beabbc9b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9\" (UID: \"911edb88-1e60-40b0-843e-7086beabbc9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9"
Nov 03 00:27:47.088649 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.088627    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e37c9d-dockercfg-47t6n\""
Nov 03 00:27:47.107486 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.107459    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk"]
Nov 03 00:27:47.119352 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.119328    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk"
Nov 03 00:27:47.131412 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.131386    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk"]
Nov 03 00:27:47.206353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.206279    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59ds2\" (UniqueName: \"kubernetes.io/projected/911edb88-1e60-40b0-843e-7086beabbc9b-kube-api-access-59ds2\") pod \"payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9\" (UID: \"911edb88-1e60-40b0-843e-7086beabbc9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9"
Nov 03 00:27:47.206562 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.206378    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8b4s\" (UniqueName: \"kubernetes.io/projected/6ab5fda5-8837-454e-9554-941f04251283-kube-api-access-m8b4s\") pod \"payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk\" (UID: \"6ab5fda5-8837-454e-9554-941f04251283\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk"
Nov 03 00:27:47.206562 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.206434    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911edb88-1e60-40b0-843e-7086beabbc9b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9\" (UID: \"911edb88-1e60-40b0-843e-7086beabbc9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9"
Nov 03 00:27:47.206562 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.206488    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab5fda5-8837-454e-9554-941f04251283-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk\" (UID: \"6ab5fda5-8837-454e-9554-941f04251283\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk"
Nov 03 00:27:47.207829 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.207794    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911edb88-1e60-40b0-843e-7086beabbc9b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9\" (UID: \"911edb88-1e60-40b0-843e-7086beabbc9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9"
Nov 03 00:27:47.268819 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.268787    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59ds2\" (UniqueName: \"kubernetes.io/projected/911edb88-1e60-40b0-843e-7086beabbc9b-kube-api-access-59ds2\") pod \"payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9\" (UID: \"911edb88-1e60-40b0-843e-7086beabbc9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9"
Nov 03 00:27:47.292219 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.292176    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85ac030e-4b72-48bb-956c-90f21ca74de1" path="/var/lib/kubelet/pods/85ac030e-4b72-48bb-956c-90f21ca74de1/volumes"
Nov 03 00:27:47.307307 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.307228    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8b4s\" (UniqueName: \"kubernetes.io/projected/6ab5fda5-8837-454e-9554-941f04251283-kube-api-access-m8b4s\") pod \"payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk\" (UID: \"6ab5fda5-8837-454e-9554-941f04251283\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk"
Nov 03 00:27:47.307307 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.307292    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab5fda5-8837-454e-9554-941f04251283-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk\" (UID: \"6ab5fda5-8837-454e-9554-941f04251283\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk"
Nov 03 00:27:47.308253 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.308226    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab5fda5-8837-454e-9554-941f04251283-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk\" (UID: \"6ab5fda5-8837-454e-9554-941f04251283\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk"
Nov 03 00:27:47.325310 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.325281    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8b4s\" (UniqueName: \"kubernetes.io/projected/6ab5fda5-8837-454e-9554-941f04251283-kube-api-access-m8b4s\") pod \"payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk\" (UID: \"6ab5fda5-8837-454e-9554-941f04251283\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk"
Nov 03 00:27:47.365213 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.365183    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9"
Nov 03 00:27:47.451458 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.451245    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk"
Nov 03 00:27:47.679332 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:27:47.679291    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911edb88_1e60_40b0_843e_7086beabbc9b.slice/crio-8a0693b45dfb4225a4300eb38e759d4461198e0d9ae7c61299152d9fd978818b WatchSource:0}: Error finding container 8a0693b45dfb4225a4300eb38e759d4461198e0d9ae7c61299152d9fd978818b: Status 404 returned error can't find the container with id 8a0693b45dfb4225a4300eb38e759d4461198e0d9ae7c61299152d9fd978818b
Nov 03 00:27:47.681110 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.681083    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9"]
Nov 03 00:27:47.704629 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.704600    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9" event={"ID":"911edb88-1e60-40b0-843e-7086beabbc9b","Type":"ContainerStarted","Data":"8a0693b45dfb4225a4300eb38e759d4461198e0d9ae7c61299152d9fd978818b"}
Nov 03 00:27:47.763029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:47.763000    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk"]
Nov 03 00:27:47.763861 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:27:47.763831    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab5fda5_8837_454e_9554_941f04251283.slice/crio-2d6f147014f1248251047e49d2b7d3d03e247f068a953c9f954977795f44592c WatchSource:0}: Error finding container 2d6f147014f1248251047e49d2b7d3d03e247f068a953c9f954977795f44592c: Status 404 returned error can't find the container with id 2d6f147014f1248251047e49d2b7d3d03e247f068a953c9f954977795f44592c
Nov 03 00:27:48.713772 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:48.713738    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk" event={"ID":"6ab5fda5-8837-454e-9554-941f04251283","Type":"ContainerStarted","Data":"3d00e032103f2ced513f798e31fbb0951eedcffc29ab403243eb762561642b0c"}
Nov 03 00:27:48.714334 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:48.713818    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk" event={"ID":"6ab5fda5-8837-454e-9554-941f04251283","Type":"ContainerStarted","Data":"2d6f147014f1248251047e49d2b7d3d03e247f068a953c9f954977795f44592c"}
Nov 03 00:27:48.714334 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:48.713841    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk"
Nov 03 00:27:48.715518 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:48.715487    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9" event={"ID":"911edb88-1e60-40b0-843e-7086beabbc9b","Type":"ContainerStarted","Data":"6f0e9929b9acd16306e01b4c07dc72becf81e8d94c2cfa7c12ae071f38543193"}
Nov 03 00:27:48.715631 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:48.715548    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9"
Nov 03 00:27:48.750286 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:48.750243    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk" podStartSLOduration=1.750229211 podStartE2EDuration="1.750229211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:27:48.74758302 +0000 UTC m=+1351.627167065" watchObservedRunningTime="2026-11-03 00:27:48.750229211 +0000 UTC m=+1351.629813254"
Nov 03 00:27:48.784346 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:48.784304    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9" podStartSLOduration=1.784291922 podStartE2EDuration="1.784291922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:27:48.780768892 +0000 UTC m=+1351.660352937" watchObservedRunningTime="2026-11-03 00:27:48.784291922 +0000 UTC m=+1351.663875965"
Nov 03 00:27:50.431283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.431247    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"]
Nov 03 00:27:50.436723 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.436699    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.459203 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.459172    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"]
Nov 03 00:27:50.462259 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.462236    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a43474-openshift-default-dockercfg-nw5kc\""
Nov 03 00:27:50.560961 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.553392    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-data\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.560961 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.553557    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ss5z\" (UniqueName: \"kubernetes.io/projected/b4f51e9c-5f2c-41df-a26f-4978e47bf980-kube-api-access-7ss5z\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.560961 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.553603    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-workload-socket\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.560961 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.553691    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-workload-certs\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.560961 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.553720    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istiod-ca-cert\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.560961 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.553765    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-envoy\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.560961 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.553806    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-podinfo\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.560961 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.553841    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-credential-socket\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.560961 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.553904    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-token\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.656039 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.655884    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ss5z\" (UniqueName: \"kubernetes.io/projected/b4f51e9c-5f2c-41df-a26f-4978e47bf980-kube-api-access-7ss5z\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.656039 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.655952    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-workload-socket\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.656039 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.656038    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-workload-certs\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.656347 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.656062    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istiod-ca-cert\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.656347 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.656130    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-envoy\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.656347 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.656194    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-podinfo\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.656347 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.656232    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-credential-socket\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.656347 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.656326    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-token\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.656646 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.656441    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-data\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.659095 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.656852    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-data\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.659095 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.656956    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istiod-ca-cert\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.659095 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.658073    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-workload-certs\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.659095 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.658218    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-workload-socket\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.659095 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.659006    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-envoy\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.662483 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.661389    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-podinfo\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.662483 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.662195    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-credential-socket\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.675867 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.675841    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ss5z\" (UniqueName: \"kubernetes.io/projected/b4f51e9c-5f2c-41df-a26f-4978e47bf980-kube-api-access-7ss5z\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.677244 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.677217    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-token\") pod \"e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") " pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:50.761122 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:50.761042    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:51.085060 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:51.085031    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"]
Nov 03 00:27:51.085363 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:27:51.085341    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f51e9c_5f2c_41df_a26f_4978e47bf980.slice/crio-962a8d3b70763ea9ae237214e3f30eeaf0b54491ab66b78d385d1c2d5a26f61f WatchSource:0}: Error finding container 962a8d3b70763ea9ae237214e3f30eeaf0b54491ab66b78d385d1c2d5a26f61f: Status 404 returned error can't find the container with id 962a8d3b70763ea9ae237214e3f30eeaf0b54491ab66b78d385d1c2d5a26f61f
Nov 03 00:27:51.087924 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:51.087876    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:27:51.088037 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:51.087967    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:27:51.088037 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:51.088012    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:27:51.742888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:51.742855    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h" event={"ID":"b4f51e9c-5f2c-41df-a26f-4978e47bf980","Type":"ContainerStarted","Data":"81e9a006c55992de198eb569c21ec23b96c9a96cfc3c317c947c754e27f334d4"}
Nov 03 00:27:51.743281 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:51.742925    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h" event={"ID":"b4f51e9c-5f2c-41df-a26f-4978e47bf980","Type":"ContainerStarted","Data":"962a8d3b70763ea9ae237214e3f30eeaf0b54491ab66b78d385d1c2d5a26f61f"}
Nov 03 00:27:51.761435 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:51.761409    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:51.804108 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:51.804058    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h" podStartSLOduration=1.804045361 podStartE2EDuration="1.804045361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:27:51.79020218 +0000 UTC m=+1354.669786223" watchObservedRunningTime="2026-11-03 00:27:51.804045361 +0000 UTC m=+1354.683629403"
Nov 03 00:27:52.766547 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:52.766516    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:53.758585 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:53.758550    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:53.759407 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:53.759383    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:27:56.137874 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:56.137842    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j"]
Nov 03 00:27:56.143213 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:56.143193    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j"
Nov 03 00:27:56.161423 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:56.161401    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j"]
Nov 03 00:27:56.166799 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:56.166768    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a43474\""
Nov 03 00:27:56.230409 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:56.223308    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a43474-dockercfg-scth9\""
Nov 03 00:27:56.334930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:56.324496    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j\" (UID: \"1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j"
Nov 03 00:27:56.334930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:56.324621    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bj9v\" (UniqueName: \"kubernetes.io/projected/1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef-kube-api-access-2bj9v\") pod \"payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j\" (UID: \"1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j"
Nov 03 00:27:56.358519 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:56.356872    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh"]
Nov 03 00:27:56.376677 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:56.376640    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh"]
Nov 03 00:27:56.376849 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:56.376737    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh"
Nov 03 00:27:56.426003 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:56.425940    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j\" (UID: \"1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j"
Nov 03 00:27:56.426137 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:56.426026    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bj9v\" (UniqueName: \"kubernetes.io/projected/1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef-kube-api-access-2bj9v\") pod \"payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j\" (UID: \"1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j"
Nov 03 00:27:56.426653 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:56.426630    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j\" (UID: \"1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j"
Nov 03 00:27:56.443296 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:56.443272    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bj9v\" (UniqueName: \"kubernetes.io/projected/1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef-kube-api-access-2bj9v\") pod \"payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j\" (UID: \"1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j"
Nov 03 00:27:56.458278 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:56.458255    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j"
Nov 03 00:27:56.527781 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:56.527748    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61049872-551b-4d49-bf01-ed17bbf2353e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh\" (UID: \"61049872-551b-4d49-bf01-ed17bbf2353e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh"
Nov 03 00:27:56.528253 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:56.528159    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmwhr\" (UniqueName: \"kubernetes.io/projected/61049872-551b-4d49-bf01-ed17bbf2353e-kube-api-access-lmwhr\") pod \"payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh\" (UID: \"61049872-551b-4d49-bf01-ed17bbf2353e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh"
Nov 03 00:27:56.629559 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:56.629528    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61049872-551b-4d49-bf01-ed17bbf2353e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh\" (UID: \"61049872-551b-4d49-bf01-ed17bbf2353e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh"
Nov 03 00:27:56.629722 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:56.629711    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmwhr\" (UniqueName: \"kubernetes.io/projected/61049872-551b-4d49-bf01-ed17bbf2353e-kube-api-access-lmwhr\") pod \"payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh\" (UID: \"61049872-551b-4d49-bf01-ed17bbf2353e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh"
Nov 03 00:27:56.630260 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:56.630237    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61049872-551b-4d49-bf01-ed17bbf2353e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh\" (UID: \"61049872-551b-4d49-bf01-ed17bbf2353e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh"
Nov 03 00:27:56.646935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:56.646887    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmwhr\" (UniqueName: \"kubernetes.io/projected/61049872-551b-4d49-bf01-ed17bbf2353e-kube-api-access-lmwhr\") pod \"payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh\" (UID: \"61049872-551b-4d49-bf01-ed17bbf2353e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh"
Nov 03 00:27:56.693018 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:56.692964    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh"
Nov 03 00:27:56.780588 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:27:56.780422    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e647aa0_04c1_4fd9_bb24_c0cc4a7d60ef.slice/crio-13ac4b73dc2b0948dfc1727807ccece19eec502578935d9ed2684cfa0f5d9f83 WatchSource:0}: Error finding container 13ac4b73dc2b0948dfc1727807ccece19eec502578935d9ed2684cfa0f5d9f83: Status 404 returned error can't find the container with id 13ac4b73dc2b0948dfc1727807ccece19eec502578935d9ed2684cfa0f5d9f83
Nov 03 00:27:56.809820 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:56.809787    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j"]
Nov 03 00:27:56.992335 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:56.992310    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh"]
Nov 03 00:27:56.992545 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:27:56.992523    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61049872_551b_4d49_bf01_ed17bbf2353e.slice/crio-12e20ee337e9a589e29c9c3a815fc3c274a9c323a021f47e3c41bea8dde1a5d6 WatchSource:0}: Error finding container 12e20ee337e9a589e29c9c3a815fc3c274a9c323a021f47e3c41bea8dde1a5d6: Status 404 returned error can't find the container with id 12e20ee337e9a589e29c9c3a815fc3c274a9c323a021f47e3c41bea8dde1a5d6
Nov 03 00:27:57.703697 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:57.703667    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6"]
Nov 03 00:27:57.736311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:57.736275    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655"]
Nov 03 00:27:57.806028 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:57.805994    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j" event={"ID":"1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef","Type":"ContainerStarted","Data":"aca9b9149b2893bbd9f615f77ffcc4682d230b354926d0d0ded7e53de305d880"}
Nov 03 00:27:57.806242 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:57.806054    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j" event={"ID":"1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef","Type":"ContainerStarted","Data":"13ac4b73dc2b0948dfc1727807ccece19eec502578935d9ed2684cfa0f5d9f83"}
Nov 03 00:27:57.806960 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:57.806845    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j"
Nov 03 00:27:57.809089 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:57.809060    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh" event={"ID":"61049872-551b-4d49-bf01-ed17bbf2353e","Type":"ContainerStarted","Data":"fb83a95b13e1a19e932e7c63e0393d100a977aa4e95c2be01ff8280191141879"}
Nov 03 00:27:57.809220 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:57.809102    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh" event={"ID":"61049872-551b-4d49-bf01-ed17bbf2353e","Type":"ContainerStarted","Data":"12e20ee337e9a589e29c9c3a815fc3c274a9c323a021f47e3c41bea8dde1a5d6"}
Nov 03 00:27:57.809220 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:57.809128    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh"
Nov 03 00:27:57.845014 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:57.844945    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j" podStartSLOduration=1.84492778 podStartE2EDuration="1.84492778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:27:57.842299867 +0000 UTC m=+1360.721883911" watchObservedRunningTime="2026-11-03 00:27:57.84492778 +0000 UTC m=+1360.724511825"
Nov 03 00:27:57.887135 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:57.887090    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh" podStartSLOduration=1.887077498 podStartE2EDuration="1.887077498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:27:57.884745132 +0000 UTC m=+1360.764329177" watchObservedRunningTime="2026-11-03 00:27:57.887077498 +0000 UTC m=+1360.766661559"
Nov 03 00:27:59.675935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:59.675747    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd"]
Nov 03 00:27:59.726932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:59.722538    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd"
Nov 03 00:27:59.727123 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:59.727008    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd"]
Nov 03 00:27:59.764327 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:59.747023    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-dda6df\""
Nov 03 00:27:59.795590 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:59.788859    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9"
Nov 03 00:27:59.795590 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:59.791490    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr"]
Nov 03 00:27:59.800165 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:59.799630    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr"
Nov 03 00:27:59.809879 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:59.809851    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk"
Nov 03 00:27:59.812358 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:59.812330    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr"]
Nov 03 00:27:59.815676 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:59.815650    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-dda6df-dockercfg-4wz8q\""
Nov 03 00:27:59.894539 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:59.894492    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqr6b\" (UniqueName: \"kubernetes.io/projected/3dc8852e-27d6-4295-8580-cb92fcea2eaf-kube-api-access-xqr6b\") pod \"payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr\" (UID: \"3dc8852e-27d6-4295-8580-cb92fcea2eaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr"
Nov 03 00:27:59.894778 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:59.894757    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc8852e-27d6-4295-8580-cb92fcea2eaf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr\" (UID: \"3dc8852e-27d6-4295-8580-cb92fcea2eaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr"
Nov 03 00:27:59.894903 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:59.894879    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/860bdb4a-93f0-45de-b939-8471e7b7aecb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd\" (UID: \"860bdb4a-93f0-45de-b939-8471e7b7aecb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd"
Nov 03 00:27:59.894996 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:59.894954    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95lgj\" (UniqueName: \"kubernetes.io/projected/860bdb4a-93f0-45de-b939-8471e7b7aecb-kube-api-access-95lgj\") pod \"payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd\" (UID: \"860bdb4a-93f0-45de-b939-8471e7b7aecb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd"
Nov 03 00:27:59.995735 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:59.995665    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqr6b\" (UniqueName: \"kubernetes.io/projected/3dc8852e-27d6-4295-8580-cb92fcea2eaf-kube-api-access-xqr6b\") pod \"payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr\" (UID: \"3dc8852e-27d6-4295-8580-cb92fcea2eaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr"
Nov 03 00:27:59.995871 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:59.995819    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc8852e-27d6-4295-8580-cb92fcea2eaf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr\" (UID: \"3dc8852e-27d6-4295-8580-cb92fcea2eaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr"
Nov 03 00:27:59.995945 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:59.995890    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/860bdb4a-93f0-45de-b939-8471e7b7aecb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd\" (UID: \"860bdb4a-93f0-45de-b939-8471e7b7aecb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd"
Nov 03 00:27:59.996002 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:59.995967    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95lgj\" (UniqueName: \"kubernetes.io/projected/860bdb4a-93f0-45de-b939-8471e7b7aecb-kube-api-access-95lgj\") pod \"payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd\" (UID: \"860bdb4a-93f0-45de-b939-8471e7b7aecb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd"
Nov 03 00:27:59.996537 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:59.996517    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc8852e-27d6-4295-8580-cb92fcea2eaf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr\" (UID: \"3dc8852e-27d6-4295-8580-cb92fcea2eaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr"
Nov 03 00:27:59.996616 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:27:59.996570    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/860bdb4a-93f0-45de-b939-8471e7b7aecb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd\" (UID: \"860bdb4a-93f0-45de-b939-8471e7b7aecb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd"
Nov 03 00:28:00.021787 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:00.021754    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqr6b\" (UniqueName: \"kubernetes.io/projected/3dc8852e-27d6-4295-8580-cb92fcea2eaf-kube-api-access-xqr6b\") pod \"payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr\" (UID: \"3dc8852e-27d6-4295-8580-cb92fcea2eaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr"
Nov 03 00:28:00.021873 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:00.021789    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95lgj\" (UniqueName: \"kubernetes.io/projected/860bdb4a-93f0-45de-b939-8471e7b7aecb-kube-api-access-95lgj\") pod \"payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd\" (UID: \"860bdb4a-93f0-45de-b939-8471e7b7aecb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd"
Nov 03 00:28:00.079544 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:00.079513    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd"
Nov 03 00:28:00.139774 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:00.139747    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr"
Nov 03 00:28:00.380392 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:00.380365    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd"]
Nov 03 00:28:00.380498 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:28:00.380392    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod860bdb4a_93f0_45de_b939_8471e7b7aecb.slice/crio-ee85960016771e2544631d199ebca89f5d6bca0d0ddfd27e089f6fed5b35f112 WatchSource:0}: Error finding container ee85960016771e2544631d199ebca89f5d6bca0d0ddfd27e089f6fed5b35f112: Status 404 returned error can't find the container with id ee85960016771e2544631d199ebca89f5d6bca0d0ddfd27e089f6fed5b35f112
Nov 03 00:28:00.459377 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:28:00.459332    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dc8852e_27d6_4295_8580_cb92fcea2eaf.slice/crio-9d03c72d451e71449969a6978ba420892d8f33ddffb5089b962e0746a470cbca WatchSource:0}: Error finding container 9d03c72d451e71449969a6978ba420892d8f33ddffb5089b962e0746a470cbca: Status 404 returned error can't find the container with id 9d03c72d451e71449969a6978ba420892d8f33ddffb5089b962e0746a470cbca
Nov 03 00:28:00.460109 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:00.460080    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr"]
Nov 03 00:28:00.854000 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:00.853964    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd" event={"ID":"860bdb4a-93f0-45de-b939-8471e7b7aecb","Type":"ContainerStarted","Data":"b6471b3426efd6cdfc9fc5c62242a5db391f3f6f369279de617390be151398fb"}
Nov 03 00:28:00.854466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:00.854024    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd" event={"ID":"860bdb4a-93f0-45de-b939-8471e7b7aecb","Type":"ContainerStarted","Data":"ee85960016771e2544631d199ebca89f5d6bca0d0ddfd27e089f6fed5b35f112"}
Nov 03 00:28:00.854755 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:00.854655    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd"
Nov 03 00:28:00.856159 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:00.856119    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr" event={"ID":"3dc8852e-27d6-4295-8580-cb92fcea2eaf","Type":"ContainerStarted","Data":"c5535c7ab052580fad527fff2a1cfe845953291a879e14dd7129fc97192e9ff9"}
Nov 03 00:28:00.856286 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:00.856197    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr"
Nov 03 00:28:00.856286 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:00.856231    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr" event={"ID":"3dc8852e-27d6-4295-8580-cb92fcea2eaf","Type":"ContainerStarted","Data":"9d03c72d451e71449969a6978ba420892d8f33ddffb5089b962e0746a470cbca"}
Nov 03 00:28:00.896956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:00.896899    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd" podStartSLOduration=1.8968865620000002 podStartE2EDuration="1.896886562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:28:00.893326754 +0000 UTC m=+1363.772910797" watchObservedRunningTime="2026-11-03 00:28:00.896886562 +0000 UTC m=+1363.776470605"
Nov 03 00:28:00.934389 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:00.934337    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr" podStartSLOduration=1.934319589 podStartE2EDuration="1.934319589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:28:00.928263256 +0000 UTC m=+1363.807847299" watchObservedRunningTime="2026-11-03 00:28:00.934319589 +0000 UTC m=+1363.813903634"
Nov 03 00:28:02.704711 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:02.704673    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6" podUID="e4cbe05e-6056-425e-b7c8-779e9ab9a6b2" containerName="payload-processing" containerID="cri-o://fb187e9637d8b0d7acd770ad24d9a7928bdd2116044a1706dc485ff37c992ce0" gracePeriod=25
Nov 03 00:28:02.737391 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:02.737357    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655" podUID="d88538c1-dccb-47da-92e9-94738e7be751" containerName="payload-pre-processing" containerID="cri-o://f9af6154cb885440dee5ea9417aeb0fad01f2b8fa41f867b0dcd170ae9c2838f" gracePeriod=25
Nov 03 00:28:02.878221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:02.877770    5012 generic.go:356] "Generic (PLEG): container finished" podID="d88538c1-dccb-47da-92e9-94738e7be751" containerID="f9af6154cb885440dee5ea9417aeb0fad01f2b8fa41f867b0dcd170ae9c2838f" exitCode=0
Nov 03 00:28:02.878221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:02.877945    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655" event={"ID":"d88538c1-dccb-47da-92e9-94738e7be751","Type":"ContainerDied","Data":"f9af6154cb885440dee5ea9417aeb0fad01f2b8fa41f867b0dcd170ae9c2838f"}
Nov 03 00:28:02.881927 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:02.880267    5012 generic.go:356] "Generic (PLEG): container finished" podID="e4cbe05e-6056-425e-b7c8-779e9ab9a6b2" containerID="fb187e9637d8b0d7acd770ad24d9a7928bdd2116044a1706dc485ff37c992ce0" exitCode=0
Nov 03 00:28:02.881927 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:02.881428    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6" event={"ID":"e4cbe05e-6056-425e-b7c8-779e9ab9a6b2","Type":"ContainerDied","Data":"fb187e9637d8b0d7acd770ad24d9a7928bdd2116044a1706dc485ff37c992ce0"}
Nov 03 00:28:03.021627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:03.021600    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6"
Nov 03 00:28:03.048844 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:03.048818    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655"
Nov 03 00:28:03.142094 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:03.142064    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4cbe05e-6056-425e-b7c8-779e9ab9a6b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4cbe05e-6056-425e-b7c8-779e9ab9a6b2-plugins-config-volume\") pod \"e4cbe05e-6056-425e-b7c8-779e9ab9a6b2\" (UID: \"e4cbe05e-6056-425e-b7c8-779e9ab9a6b2\") "
Nov 03 00:28:03.142259 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:03.142101    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d88538c1-dccb-47da-92e9-94738e7be751-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d88538c1-dccb-47da-92e9-94738e7be751-plugins-config-volume\") pod \"d88538c1-dccb-47da-92e9-94738e7be751\" (UID: \"d88538c1-dccb-47da-92e9-94738e7be751\") "
Nov 03 00:28:03.142259 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:03.142220    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d88538c1-dccb-47da-92e9-94738e7be751-kube-api-access-f7mc2\" (UniqueName: \"kubernetes.io/projected/d88538c1-dccb-47da-92e9-94738e7be751-kube-api-access-f7mc2\") pod \"d88538c1-dccb-47da-92e9-94738e7be751\" (UID: \"d88538c1-dccb-47da-92e9-94738e7be751\") "
Nov 03 00:28:03.142407 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:03.142393    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4cbe05e-6056-425e-b7c8-779e9ab9a6b2-kube-api-access-vsd2b\" (UniqueName: \"kubernetes.io/projected/e4cbe05e-6056-425e-b7c8-779e9ab9a6b2-kube-api-access-vsd2b\") pod \"e4cbe05e-6056-425e-b7c8-779e9ab9a6b2\" (UID: \"e4cbe05e-6056-425e-b7c8-779e9ab9a6b2\") "
Nov 03 00:28:03.142566 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:03.142538    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4cbe05e-6056-425e-b7c8-779e9ab9a6b2-plugins-config-volume" pod "e4cbe05e-6056-425e-b7c8-779e9ab9a6b2" (UID: "e4cbe05e-6056-425e-b7c8-779e9ab9a6b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:28:03.142685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:03.142564    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d88538c1-dccb-47da-92e9-94738e7be751-plugins-config-volume" pod "d88538c1-dccb-47da-92e9-94738e7be751" (UID: "d88538c1-dccb-47da-92e9-94738e7be751"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:28:03.142921 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:03.142891    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4cbe05e-6056-425e-b7c8-779e9ab9a6b2-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:03.142972 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:03.142927    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d88538c1-dccb-47da-92e9-94738e7be751-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:03.144304 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:03.144283    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d88538c1-dccb-47da-92e9-94738e7be751-kube-api-access-f7mc2" pod "d88538c1-dccb-47da-92e9-94738e7be751" (UID: "d88538c1-dccb-47da-92e9-94738e7be751"). InnerVolumeSpecName "kube-api-access-f7mc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:03.144378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:03.144298    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4cbe05e-6056-425e-b7c8-779e9ab9a6b2-kube-api-access-vsd2b" pod "e4cbe05e-6056-425e-b7c8-779e9ab9a6b2" (UID: "e4cbe05e-6056-425e-b7c8-779e9ab9a6b2"). InnerVolumeSpecName "kube-api-access-vsd2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:03.243875 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:03.243800    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7mc2\" (UniqueName: \"kubernetes.io/projected/d88538c1-dccb-47da-92e9-94738e7be751-kube-api-access-f7mc2\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:03.243875 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:03.243834    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsd2b\" (UniqueName: \"kubernetes.io/projected/e4cbe05e-6056-425e-b7c8-779e9ab9a6b2-kube-api-access-vsd2b\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:03.892723 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:03.892692    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655"
Nov 03 00:28:03.893198 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:03.892688    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655" event={"ID":"d88538c1-dccb-47da-92e9-94738e7be751","Type":"ContainerDied","Data":"0e16d51d625a20ee2c8faafcc3e9e8a860adf202cd718bcda795c8d935c6957d"}
Nov 03 00:28:03.893198 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:03.892831    5012 scope.go:122] "RemoveContainer" containerID="f9af6154cb885440dee5ea9417aeb0fad01f2b8fa41f867b0dcd170ae9c2838f"
Nov 03 00:28:03.894722 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:03.894703    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6"
Nov 03 00:28:03.894722 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:03.894708    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6" event={"ID":"e4cbe05e-6056-425e-b7c8-779e9ab9a6b2","Type":"ContainerDied","Data":"7d42d59c09d4315ec5715f19907f30734616ede2859adb64b719b44bbdc0b7e0"}
Nov 03 00:28:03.904270 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:03.904246    5012 scope.go:122] "RemoveContainer" containerID="fb187e9637d8b0d7acd770ad24d9a7928bdd2116044a1706dc485ff37c992ce0"
Nov 03 00:28:03.942062 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:03.942035    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655"]
Nov 03 00:28:03.951445 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:03.951424    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-27490b-df699c78c-tl655"]
Nov 03 00:28:03.974660 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:03.974614    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6"]
Nov 03 00:28:03.978587 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:03.978568    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-27490b-5d5c658567-4qsv6"]
Nov 03 00:28:05.245736 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:05.245705    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d88538c1-dccb-47da-92e9-94738e7be751" path="/var/lib/kubelet/pods/d88538c1-dccb-47da-92e9-94738e7be751/volumes"
Nov 03 00:28:05.246236 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:05.246136    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4cbe05e-6056-425e-b7c8-779e9ab9a6b2" path="/var/lib/kubelet/pods/e4cbe05e-6056-425e-b7c8-779e9ab9a6b2/volumes"
Nov 03 00:28:07.364157 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:28:07.364121    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:28:08.819840 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:08.819810    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh"
Nov 03 00:28:09.843046 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:09.843014    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j"
Nov 03 00:28:11.594774 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:11.594733    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"]
Nov 03 00:28:11.595265 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:11.595084    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76" podUID="94d99e83-8a4e-4d07-a350-571f916a764f" containerName="istio-proxy" containerID="cri-o://cadd1a11dada131dcdd96a3f0cd33734ebc9deef6ab5aaae3c8b22bbaf1f874b" gracePeriod=30
Nov 03 00:28:11.605067 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:11.603215    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8"]
Nov 03 00:28:11.627501 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:11.627478    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2"]
Nov 03 00:28:11.891604 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:11.891558    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr"
Nov 03 00:28:12.884332 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:12.884303    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd"
Nov 03 00:28:16.604150 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:16.604107    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8" podUID="0065a2de-0be6-464b-b917-348a84615c51" containerName="payload-processing" containerID="cri-o://a384cf5e99290841f17247aacc0845dd68543ddfc5fd631048fa76828e0f26b6" gracePeriod=25
Nov 03 00:28:16.627816 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:16.627784    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2" podUID="30042d87-468b-416c-955f-685f9b7d0994" containerName="payload-pre-processing" containerID="cri-o://63766fe2aec56d5132efd690b59f6ca30c937f17c70f17bd0f17c515427abe37" gracePeriod=25
Nov 03 00:28:16.976985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:16.976957    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8"
Nov 03 00:28:17.049649 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.049620    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2"
Nov 03 00:28:17.055060 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.055011    5012 generic.go:356] "Generic (PLEG): container finished" podID="30042d87-468b-416c-955f-685f9b7d0994" containerID="63766fe2aec56d5132efd690b59f6ca30c937f17c70f17bd0f17c515427abe37" exitCode=0
Nov 03 00:28:17.055475 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.055063    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2" event={"ID":"30042d87-468b-416c-955f-685f9b7d0994","Type":"ContainerDied","Data":"63766fe2aec56d5132efd690b59f6ca30c937f17c70f17bd0f17c515427abe37"}
Nov 03 00:28:17.055767 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.055721    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2" event={"ID":"30042d87-468b-416c-955f-685f9b7d0994","Type":"ContainerDied","Data":"3e8cc5ba1acc16a2068c12ab1ea093ab93e7b848eae2c08c2608bd744eae7f52"}
Nov 03 00:28:17.056013 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.055983    5012 scope.go:122] "RemoveContainer" containerID="63766fe2aec56d5132efd690b59f6ca30c937f17c70f17bd0f17c515427abe37"
Nov 03 00:28:17.060393 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.060356    5012 generic.go:356] "Generic (PLEG): container finished" podID="94d99e83-8a4e-4d07-a350-571f916a764f" containerID="cadd1a11dada131dcdd96a3f0cd33734ebc9deef6ab5aaae3c8b22bbaf1f874b" exitCode=0
Nov 03 00:28:17.060502 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.060486    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76" event={"ID":"94d99e83-8a4e-4d07-a350-571f916a764f","Type":"ContainerDied","Data":"cadd1a11dada131dcdd96a3f0cd33734ebc9deef6ab5aaae3c8b22bbaf1f874b"}
Nov 03 00:28:17.062276 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.062228    5012 generic.go:356] "Generic (PLEG): container finished" podID="0065a2de-0be6-464b-b917-348a84615c51" containerID="a384cf5e99290841f17247aacc0845dd68543ddfc5fd631048fa76828e0f26b6" exitCode=0
Nov 03 00:28:17.062522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.062485    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8" event={"ID":"0065a2de-0be6-464b-b917-348a84615c51","Type":"ContainerDied","Data":"a384cf5e99290841f17247aacc0845dd68543ddfc5fd631048fa76828e0f26b6"}
Nov 03 00:28:17.062617 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.062547    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8" event={"ID":"0065a2de-0be6-464b-b917-348a84615c51","Type":"ContainerDied","Data":"563d7f54b57a228e30d02c888551ecd6c894e5192996fbcdf238939c329b11c2"}
Nov 03 00:28:17.062683 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.062641    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8"
Nov 03 00:28:17.063228 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.063202    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:28:17.067091 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.067068    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-istio-envoy\") pod \"94d99e83-8a4e-4d07-a350-571f916a764f\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") "
Nov 03 00:28:17.067200 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.067115    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-credential-socket\") pod \"94d99e83-8a4e-4d07-a350-571f916a764f\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") "
Nov 03 00:28:17.067200 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.067176    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94d99e83-8a4e-4d07-a350-571f916a764f-kube-api-access-sckkd\" (UniqueName: \"kubernetes.io/projected/94d99e83-8a4e-4d07-a350-571f916a764f-kube-api-access-sckkd\") pod \"94d99e83-8a4e-4d07-a350-571f916a764f\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") "
Nov 03 00:28:17.067321 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.067215    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30042d87-468b-416c-955f-685f9b7d0994-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30042d87-468b-416c-955f-685f9b7d0994-plugins-config-volume\") pod \"30042d87-468b-416c-955f-685f9b7d0994\" (UID: \"30042d87-468b-416c-955f-685f9b7d0994\") "
Nov 03 00:28:17.067394 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.067368    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-workload-certs\") pod \"94d99e83-8a4e-4d07-a350-571f916a764f\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") "
Nov 03 00:28:17.067470 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.067453    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0065a2de-0be6-464b-b917-348a84615c51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0065a2de-0be6-464b-b917-348a84615c51-plugins-config-volume\") pod \"0065a2de-0be6-464b-b917-348a84615c51\" (UID: \"0065a2de-0be6-464b-b917-348a84615c51\") "
Nov 03 00:28:17.067548 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.067531    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94d99e83-8a4e-4d07-a350-571f916a764f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d99e83-8a4e-4d07-a350-571f916a764f-istiod-ca-cert\") pod \"94d99e83-8a4e-4d07-a350-571f916a764f\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") "
Nov 03 00:28:17.067610 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.067593    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-workload-certs" pod "94d99e83-8a4e-4d07-a350-571f916a764f" (UID: "94d99e83-8a4e-4d07-a350-571f916a764f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:17.067673 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.067620    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30042d87-468b-416c-955f-685f9b7d0994-kube-api-access-p9mcl\" (UniqueName: \"kubernetes.io/projected/30042d87-468b-416c-955f-685f9b7d0994-kube-api-access-p9mcl\") pod \"30042d87-468b-416c-955f-685f9b7d0994\" (UID: \"30042d87-468b-416c-955f-685f9b7d0994\") "
Nov 03 00:28:17.067731 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.067671    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94d99e83-8a4e-4d07-a350-571f916a764f-istio-token\" (UniqueName: \"kubernetes.io/projected/94d99e83-8a4e-4d07-a350-571f916a764f-istio-token\") pod \"94d99e83-8a4e-4d07-a350-571f916a764f\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") "
Nov 03 00:28:17.067731 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.067698    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-workload-socket\") pod \"94d99e83-8a4e-4d07-a350-571f916a764f\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") "
Nov 03 00:28:17.067835 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.067756    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0065a2de-0be6-464b-b917-348a84615c51-kube-api-access-2p5hb\" (UniqueName: \"kubernetes.io/projected/0065a2de-0be6-464b-b917-348a84615c51-kube-api-access-2p5hb\") pod \"0065a2de-0be6-464b-b917-348a84615c51\" (UID: \"0065a2de-0be6-464b-b917-348a84615c51\") "
Nov 03 00:28:17.067835 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.067643    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30042d87-468b-416c-955f-685f9b7d0994-plugins-config-volume" pod "30042d87-468b-416c-955f-685f9b7d0994" (UID: "30042d87-468b-416c-955f-685f9b7d0994"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:28:17.067835 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.067767    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-credential-socket" pod "94d99e83-8a4e-4d07-a350-571f916a764f" (UID: "94d99e83-8a4e-4d07-a350-571f916a764f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:17.067835 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.067787    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-istio-data\") pod \"94d99e83-8a4e-4d07-a350-571f916a764f\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") "
Nov 03 00:28:17.067835 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.067817    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94d99e83-8a4e-4d07-a350-571f916a764f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d99e83-8a4e-4d07-a350-571f916a764f-istio-podinfo\") pod \"94d99e83-8a4e-4d07-a350-571f916a764f\" (UID: \"94d99e83-8a4e-4d07-a350-571f916a764f\") "
Nov 03 00:28:17.068137 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.067959    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0065a2de-0be6-464b-b917-348a84615c51-plugins-config-volume" pod "0065a2de-0be6-464b-b917-348a84615c51" (UID: "0065a2de-0be6-464b-b917-348a84615c51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:28:17.068137 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.068061    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d99e83-8a4e-4d07-a350-571f916a764f-istiod-ca-cert" pod "94d99e83-8a4e-4d07-a350-571f916a764f" (UID: "94d99e83-8a4e-4d07-a350-571f916a764f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:28:17.068404 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.068378    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-workload-socket" pod "94d99e83-8a4e-4d07-a350-571f916a764f" (UID: "94d99e83-8a4e-4d07-a350-571f916a764f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:17.068504 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.068482    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30042d87-468b-416c-955f-685f9b7d0994-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:17.068655 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.068631    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-istio-data" pod "94d99e83-8a4e-4d07-a350-571f916a764f" (UID: "94d99e83-8a4e-4d07-a350-571f916a764f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:17.068719 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.068637    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:17.068719 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.068703    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0065a2de-0be6-464b-b917-348a84615c51-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:17.068824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.068722    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d99e83-8a4e-4d07-a350-571f916a764f-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:17.068824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.068740    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:17.068824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.068773    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:17.071803 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.071773    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d99e83-8a4e-4d07-a350-571f916a764f-kube-api-access-sckkd" pod "94d99e83-8a4e-4d07-a350-571f916a764f" (UID: "94d99e83-8a4e-4d07-a350-571f916a764f"). InnerVolumeSpecName "kube-api-access-sckkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:17.071922 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.071816    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30042d87-468b-416c-955f-685f9b7d0994-kube-api-access-p9mcl" pod "30042d87-468b-416c-955f-685f9b7d0994" (UID: "30042d87-468b-416c-955f-685f9b7d0994"). InnerVolumeSpecName "kube-api-access-p9mcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:17.072082 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.072054    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94d99e83-8a4e-4d07-a350-571f916a764f-istio-podinfo" pod "94d99e83-8a4e-4d07-a350-571f916a764f" (UID: "94d99e83-8a4e-4d07-a350-571f916a764f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:28:17.072488 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.072468    5012 scope.go:122] "RemoveContainer" containerID="63766fe2aec56d5132efd690b59f6ca30c937f17c70f17bd0f17c515427abe37"
Nov 03 00:28:17.072770 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:28:17.072743    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63766fe2aec56d5132efd690b59f6ca30c937f17c70f17bd0f17c515427abe37\": container with ID starting with 63766fe2aec56d5132efd690b59f6ca30c937f17c70f17bd0f17c515427abe37 not found: ID does not exist" containerID="63766fe2aec56d5132efd690b59f6ca30c937f17c70f17bd0f17c515427abe37"
Nov 03 00:28:17.072857 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.072775    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63766fe2aec56d5132efd690b59f6ca30c937f17c70f17bd0f17c515427abe37"} err="failed to get container status \"63766fe2aec56d5132efd690b59f6ca30c937f17c70f17bd0f17c515427abe37\": rpc error: code = NotFound desc = could not find container \"63766fe2aec56d5132efd690b59f6ca30c937f17c70f17bd0f17c515427abe37\": container with ID starting with 63766fe2aec56d5132efd690b59f6ca30c937f17c70f17bd0f17c515427abe37 not found: ID does not exist"
Nov 03 00:28:17.072857 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.072797    5012 scope.go:122] "RemoveContainer" containerID="9dd6325444308fcad6adba4fbf8610961bdaec6e4291a280cbec5f1b9b731ada"
Nov 03 00:28:17.072998 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.072974    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0065a2de-0be6-464b-b917-348a84615c51-kube-api-access-2p5hb" pod "0065a2de-0be6-464b-b917-348a84615c51" (UID: "0065a2de-0be6-464b-b917-348a84615c51"). InnerVolumeSpecName "kube-api-access-2p5hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:17.073278 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.073253    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d99e83-8a4e-4d07-a350-571f916a764f-istio-token" pod "94d99e83-8a4e-4d07-a350-571f916a764f" (UID: "94d99e83-8a4e-4d07-a350-571f916a764f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:17.089036 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.086030    5012 scope.go:122] "RemoveContainer" containerID="a384cf5e99290841f17247aacc0845dd68543ddfc5fd631048fa76828e0f26b6"
Nov 03 00:28:17.103460 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.102624    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-istio-envoy" pod "94d99e83-8a4e-4d07-a350-571f916a764f" (UID: "94d99e83-8a4e-4d07-a350-571f916a764f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:17.104580 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.104548    5012 scope.go:122] "RemoveContainer" containerID="a384cf5e99290841f17247aacc0845dd68543ddfc5fd631048fa76828e0f26b6"
Nov 03 00:28:17.104976 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:28:17.104904    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a384cf5e99290841f17247aacc0845dd68543ddfc5fd631048fa76828e0f26b6\": container with ID starting with a384cf5e99290841f17247aacc0845dd68543ddfc5fd631048fa76828e0f26b6 not found: ID does not exist" containerID="a384cf5e99290841f17247aacc0845dd68543ddfc5fd631048fa76828e0f26b6"
Nov 03 00:28:17.105086 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.104985    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a384cf5e99290841f17247aacc0845dd68543ddfc5fd631048fa76828e0f26b6"} err="failed to get container status \"a384cf5e99290841f17247aacc0845dd68543ddfc5fd631048fa76828e0f26b6\": rpc error: code = NotFound desc = could not find container \"a384cf5e99290841f17247aacc0845dd68543ddfc5fd631048fa76828e0f26b6\": container with ID starting with a384cf5e99290841f17247aacc0845dd68543ddfc5fd631048fa76828e0f26b6 not found: ID does not exist"
Nov 03 00:28:17.170359 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.170280    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sckkd\" (UniqueName: \"kubernetes.io/projected/94d99e83-8a4e-4d07-a350-571f916a764f-kube-api-access-sckkd\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:17.170359 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.170309    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9mcl\" (UniqueName: \"kubernetes.io/projected/30042d87-468b-416c-955f-685f9b7d0994-kube-api-access-p9mcl\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:17.170359 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.170322    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d99e83-8a4e-4d07-a350-571f916a764f-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:17.170359 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.170335    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p5hb\" (UniqueName: \"kubernetes.io/projected/0065a2de-0be6-464b-b917-348a84615c51-kube-api-access-2p5hb\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:17.170359 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.170348    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:17.170359 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.170360    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d99e83-8a4e-4d07-a350-571f916a764f-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:17.170656 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.170371    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d99e83-8a4e-4d07-a350-571f916a764f-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:17.410175 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.410147    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8"]
Nov 03 00:28:17.414066 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:17.414042    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1d29dd-c5c47bd4f-ljcb8"]
Nov 03 00:28:18.072277 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.072191    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"
Nov 03 00:28:18.072277 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.072214    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76" event={"ID":"94d99e83-8a4e-4d07-a350-571f916a764f","Type":"ContainerDied","Data":"aa2dd128558c7d361539f5de018c6da7c4d8eaa153f7cfdd1eaae572c1758b97"}
Nov 03 00:28:18.072792 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.072281    5012 scope.go:122] "RemoveContainer" containerID="cadd1a11dada131dcdd96a3f0cd33734ebc9deef6ab5aaae3c8b22bbaf1f874b"
Nov 03 00:28:18.074780 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.074752    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2"
Nov 03 00:28:18.164041 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.164010    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"]
Nov 03 00:28:18.167840 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.167816    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d43409-openshift-default-6ccc9fc775-vkp76"]
Nov 03 00:28:18.189848 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.189819    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2"]
Nov 03 00:28:18.193402 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.193382    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1d29dd-7867896fff-4rcz2"]
Nov 03 00:28:18.578642 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.578610    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4"]
Nov 03 00:28:18.579331 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.579309    5012 cpu_manager.go:414] "containerMap: removing container" podUID="d88538c1-dccb-47da-92e9-94738e7be751" containerName="payload-pre-processing"
Nov 03 00:28:18.579331 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.579329    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d88538c1-dccb-47da-92e9-94738e7be751" containerName="payload-pre-processing"
Nov 03 00:28:18.579522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.579353    5012 cpu_manager.go:414] "containerMap: removing container" podUID="94d99e83-8a4e-4d07-a350-571f916a764f" containerName="istio-proxy"
Nov 03 00:28:18.579522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.579361    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d99e83-8a4e-4d07-a350-571f916a764f" containerName="istio-proxy"
Nov 03 00:28:18.579522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.579383    5012 cpu_manager.go:414] "containerMap: removing container" podUID="94d99e83-8a4e-4d07-a350-571f916a764f" containerName="istio-proxy"
Nov 03 00:28:18.579522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.579390    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d99e83-8a4e-4d07-a350-571f916a764f" containerName="istio-proxy"
Nov 03 00:28:18.579522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.579419    5012 cpu_manager.go:414] "containerMap: removing container" podUID="30042d87-468b-416c-955f-685f9b7d0994" containerName="payload-pre-processing"
Nov 03 00:28:18.579522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.579427    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30042d87-468b-416c-955f-685f9b7d0994" containerName="payload-pre-processing"
Nov 03 00:28:18.579522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.579444    5012 cpu_manager.go:414] "containerMap: removing container" podUID="e4cbe05e-6056-425e-b7c8-779e9ab9a6b2" containerName="payload-processing"
Nov 03 00:28:18.579522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.579450    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cbe05e-6056-425e-b7c8-779e9ab9a6b2" containerName="payload-processing"
Nov 03 00:28:18.579522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.579461    5012 cpu_manager.go:414] "containerMap: removing container" podUID="0065a2de-0be6-464b-b917-348a84615c51" containerName="payload-processing"
Nov 03 00:28:18.579522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.579467    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0065a2de-0be6-464b-b917-348a84615c51" containerName="payload-processing"
Nov 03 00:28:18.580046 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.579648    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="30042d87-468b-416c-955f-685f9b7d0994" containerName="payload-pre-processing"
Nov 03 00:28:18.580046 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.579663    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cbe05e-6056-425e-b7c8-779e9ab9a6b2" containerName="payload-processing"
Nov 03 00:28:18.580046 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.579677    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="0065a2de-0be6-464b-b917-348a84615c51" containerName="payload-processing"
Nov 03 00:28:18.580046 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.579695    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="d88538c1-dccb-47da-92e9-94738e7be751" containerName="payload-pre-processing"
Nov 03 00:28:18.580046 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.579710    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d99e83-8a4e-4d07-a350-571f916a764f" containerName="istio-proxy"
Nov 03 00:28:18.580046 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.579722    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d99e83-8a4e-4d07-a350-571f916a764f" containerName="istio-proxy"
Nov 03 00:28:18.585486 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.585464    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4"
Nov 03 00:28:18.586209 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.586189    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9"]
Nov 03 00:28:18.590657 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.590639    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9"
Nov 03 00:28:18.607900 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.607872    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4"]
Nov 03 00:28:18.610306 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.610288    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-54f6d8\""
Nov 03 00:28:18.617523 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.617478    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9"]
Nov 03 00:28:18.650834 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.650803    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj"]
Nov 03 00:28:18.656476 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.656453    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj"
Nov 03 00:28:18.670787 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.670764    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a43474-serving-cert\""
Nov 03 00:28:18.671648 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.671626    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-54f6d8-dockercfg-qsbt2\""
Nov 03 00:28:18.674962 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.674936    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj"]
Nov 03 00:28:18.687804 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.687780    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8503d3f7-ece8-4113-8888-dd5cfbe73073-metrics-certs\") pod \"maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4\" (UID: \"8503d3f7-ece8-4113-8888-dd5cfbe73073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4"
Nov 03 00:28:18.687900 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.687818    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8503d3f7-ece8-4113-8888-dd5cfbe73073-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4\" (UID: \"8503d3f7-ece8-4113-8888-dd5cfbe73073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4"
Nov 03 00:28:18.688007 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.687987    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hc8l\" (UniqueName: \"kubernetes.io/projected/3c081efc-84f9-4246-abbf-947475128655-kube-api-access-7hc8l\") pod \"payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj\" (UID: \"3c081efc-84f9-4246-abbf-947475128655\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj"
Nov 03 00:28:18.688083 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.688067    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxfnb\" (UniqueName: \"kubernetes.io/projected/8503d3f7-ece8-4113-8888-dd5cfbe73073-kube-api-access-zxfnb\") pod \"maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4\" (UID: \"8503d3f7-ece8-4113-8888-dd5cfbe73073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4"
Nov 03 00:28:18.688119 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.688108    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8503d3f7-ece8-4113-8888-dd5cfbe73073-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4\" (UID: \"8503d3f7-ece8-4113-8888-dd5cfbe73073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4"
Nov 03 00:28:18.688153 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.688130    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c081efc-84f9-4246-abbf-947475128655-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj\" (UID: \"3c081efc-84f9-4246-abbf-947475128655\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj"
Nov 03 00:28:18.688282 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.688264    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72pd5\" (UniqueName: \"kubernetes.io/projected/09fa3da2-974b-4530-aed4-d64a3223ccbc-kube-api-access-72pd5\") pod \"payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9\" (UID: \"09fa3da2-974b-4530-aed4-d64a3223ccbc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9"
Nov 03 00:28:18.688364 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.688297    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09fa3da2-974b-4530-aed4-d64a3223ccbc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9\" (UID: \"09fa3da2-974b-4530-aed4-d64a3223ccbc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9"
Nov 03 00:28:18.738472 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.738445    5012 scope.go:122] "RemoveContainer" containerID="0f422e1940b613f10f4b17824f07b7d8dc086b7a404940bfb67715b32012d5d5"
Nov 03 00:28:18.750699 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.750680    5012 scope.go:122] "RemoveContainer" containerID="6e66032a415d09bfd1f61c2fce255e8af3fea1d92a68024b2d419880acaeed97"
Nov 03 00:28:18.789799 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.789777    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8503d3f7-ece8-4113-8888-dd5cfbe73073-metrics-certs\") pod \"maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4\" (UID: \"8503d3f7-ece8-4113-8888-dd5cfbe73073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4"
Nov 03 00:28:18.789866 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.789817    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8503d3f7-ece8-4113-8888-dd5cfbe73073-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4\" (UID: \"8503d3f7-ece8-4113-8888-dd5cfbe73073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4"
Nov 03 00:28:18.789938 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.789880    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hc8l\" (UniqueName: \"kubernetes.io/projected/3c081efc-84f9-4246-abbf-947475128655-kube-api-access-7hc8l\") pod \"payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj\" (UID: \"3c081efc-84f9-4246-abbf-947475128655\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj"
Nov 03 00:28:18.790005 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.789955    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxfnb\" (UniqueName: \"kubernetes.io/projected/8503d3f7-ece8-4113-8888-dd5cfbe73073-kube-api-access-zxfnb\") pod \"maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4\" (UID: \"8503d3f7-ece8-4113-8888-dd5cfbe73073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4"
Nov 03 00:28:18.790005 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.789981    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8503d3f7-ece8-4113-8888-dd5cfbe73073-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4\" (UID: \"8503d3f7-ece8-4113-8888-dd5cfbe73073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4"
Nov 03 00:28:18.790110 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.790005    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c081efc-84f9-4246-abbf-947475128655-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj\" (UID: \"3c081efc-84f9-4246-abbf-947475128655\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj"
Nov 03 00:28:18.790110 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.790062    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72pd5\" (UniqueName: \"kubernetes.io/projected/09fa3da2-974b-4530-aed4-d64a3223ccbc-kube-api-access-72pd5\") pod \"payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9\" (UID: \"09fa3da2-974b-4530-aed4-d64a3223ccbc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9"
Nov 03 00:28:18.790110 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.790084    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09fa3da2-974b-4530-aed4-d64a3223ccbc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9\" (UID: \"09fa3da2-974b-4530-aed4-d64a3223ccbc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9"
Nov 03 00:28:18.790662 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.790635    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8503d3f7-ece8-4113-8888-dd5cfbe73073-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4\" (UID: \"8503d3f7-ece8-4113-8888-dd5cfbe73073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4"
Nov 03 00:28:18.790869 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.790848    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09fa3da2-974b-4530-aed4-d64a3223ccbc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9\" (UID: \"09fa3da2-974b-4530-aed4-d64a3223ccbc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9"
Nov 03 00:28:18.790966 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.790942    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c081efc-84f9-4246-abbf-947475128655-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj\" (UID: \"3c081efc-84f9-4246-abbf-947475128655\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj"
Nov 03 00:28:18.792204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.792185    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8503d3f7-ece8-4113-8888-dd5cfbe73073-metrics-certs\") pod \"maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4\" (UID: \"8503d3f7-ece8-4113-8888-dd5cfbe73073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4"
Nov 03 00:28:18.793221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.793197    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8503d3f7-ece8-4113-8888-dd5cfbe73073-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4\" (UID: \"8503d3f7-ece8-4113-8888-dd5cfbe73073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4"
Nov 03 00:28:18.809771 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.809748    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxfnb\" (UniqueName: \"kubernetes.io/projected/8503d3f7-ece8-4113-8888-dd5cfbe73073-kube-api-access-zxfnb\") pod \"maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4\" (UID: \"8503d3f7-ece8-4113-8888-dd5cfbe73073\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4"
Nov 03 00:28:18.809840 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.809767    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hc8l\" (UniqueName: \"kubernetes.io/projected/3c081efc-84f9-4246-abbf-947475128655-kube-api-access-7hc8l\") pod \"payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj\" (UID: \"3c081efc-84f9-4246-abbf-947475128655\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj"
Nov 03 00:28:18.809840 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.809766    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72pd5\" (UniqueName: \"kubernetes.io/projected/09fa3da2-974b-4530-aed4-d64a3223ccbc-kube-api-access-72pd5\") pod \"payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9\" (UID: \"09fa3da2-974b-4530-aed4-d64a3223ccbc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9"
Nov 03 00:28:18.903746 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.901555    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4"
Nov 03 00:28:18.913965 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.910238    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9"
Nov 03 00:28:18.977813 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:18.977209    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj"
Nov 03 00:28:19.245700 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:19.245668    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0065a2de-0be6-464b-b917-348a84615c51" path="/var/lib/kubelet/pods/0065a2de-0be6-464b-b917-348a84615c51/volumes"
Nov 03 00:28:19.246142 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:19.246121    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30042d87-468b-416c-955f-685f9b7d0994" path="/var/lib/kubelet/pods/30042d87-468b-416c-955f-685f9b7d0994/volumes"
Nov 03 00:28:19.246600 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:19.246585    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94d99e83-8a4e-4d07-a350-571f916a764f" path="/var/lib/kubelet/pods/94d99e83-8a4e-4d07-a350-571f916a764f/volumes"
Nov 03 00:28:19.271733 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:19.271704    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4"]
Nov 03 00:28:19.271939 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:28:19.271888    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8503d3f7_ece8_4113_8888_dd5cfbe73073.slice/crio-6748e02b5430f983d27710cb2e62605ae17d4a026d05fcff116ee44bad8846c2 WatchSource:0}: Error finding container 6748e02b5430f983d27710cb2e62605ae17d4a026d05fcff116ee44bad8846c2: Status 404 returned error can't find the container with id 6748e02b5430f983d27710cb2e62605ae17d4a026d05fcff116ee44bad8846c2
Nov 03 00:28:19.292423 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:28:19.292403    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09fa3da2_974b_4530_aed4_d64a3223ccbc.slice/crio-475adbda1ab6dc21af67d25233fbc754a2e6debc3cda3b64d35b9bda7c9a1bb6 WatchSource:0}: Error finding container 475adbda1ab6dc21af67d25233fbc754a2e6debc3cda3b64d35b9bda7c9a1bb6: Status 404 returned error can't find the container with id 475adbda1ab6dc21af67d25233fbc754a2e6debc3cda3b64d35b9bda7c9a1bb6
Nov 03 00:28:19.292780 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:19.292752    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9"]
Nov 03 00:28:19.314882 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:19.314861    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj"]
Nov 03 00:28:19.315169 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:28:19.315149    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c081efc_84f9_4246_abbf_947475128655.slice/crio-08c28b91040653cc4bdf2116e410b56595fb13e7a556979c7ab9a605568d879a WatchSource:0}: Error finding container 08c28b91040653cc4bdf2116e410b56595fb13e7a556979c7ab9a605568d879a: Status 404 returned error can't find the container with id 08c28b91040653cc4bdf2116e410b56595fb13e7a556979c7ab9a605568d879a
Nov 03 00:28:20.095530 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:20.095443    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj" event={"ID":"3c081efc-84f9-4246-abbf-947475128655","Type":"ContainerStarted","Data":"c2216f4da2e346de44a6bc85776f8d3d4969287fa827bda9e6d114314939f552"}
Nov 03 00:28:20.095530 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:20.095502    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj" event={"ID":"3c081efc-84f9-4246-abbf-947475128655","Type":"ContainerStarted","Data":"08c28b91040653cc4bdf2116e410b56595fb13e7a556979c7ab9a605568d879a"}
Nov 03 00:28:20.095765 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:20.095537    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj"
Nov 03 00:28:20.097214 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:20.097184    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4" event={"ID":"8503d3f7-ece8-4113-8888-dd5cfbe73073","Type":"ContainerStarted","Data":"6bccfb823cbee7d92f87dc3b9375fa646725117b956c9f290c5695d6ba21f69b"}
Nov 03 00:28:20.097317 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:20.097234    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4" event={"ID":"8503d3f7-ece8-4113-8888-dd5cfbe73073","Type":"ContainerStarted","Data":"6748e02b5430f983d27710cb2e62605ae17d4a026d05fcff116ee44bad8846c2"}
Nov 03 00:28:20.097317 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:20.097279    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4"
Nov 03 00:28:20.098863 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:20.098839    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9" event={"ID":"09fa3da2-974b-4530-aed4-d64a3223ccbc","Type":"ContainerStarted","Data":"695c2d16d9242fa99e8cbfb2b565396a80681b36b958c54cf3c8d5d901a9ac1b"}
Nov 03 00:28:20.098979 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:20.098879    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9" event={"ID":"09fa3da2-974b-4530-aed4-d64a3223ccbc","Type":"ContainerStarted","Data":"475adbda1ab6dc21af67d25233fbc754a2e6debc3cda3b64d35b9bda7c9a1bb6"}
Nov 03 00:28:20.098979 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:20.098901    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9"
Nov 03 00:28:20.135886 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:20.135842    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj" podStartSLOduration=2.135832161 podStartE2EDuration="2.135832161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:28:20.131814509 +0000 UTC m=+1383.011398551" watchObservedRunningTime="2026-11-03 00:28:20.135832161 +0000 UTC m=+1383.015416203"
Nov 03 00:28:20.170026 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:20.169987    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9" podStartSLOduration=2.16997637 podStartE2EDuration="2.16997637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:28:20.16777057 +0000 UTC m=+1383.047354614" watchObservedRunningTime="2026-11-03 00:28:20.16997637 +0000 UTC m=+1383.049560445"
Nov 03 00:28:20.205207 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:20.205172    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4" podStartSLOduration=1.869033535 podStartE2EDuration="2.205162575s" totalImagesPullingTime="336.12904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:28:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:28:20.202263425 +0000 UTC m=+1383.081847470" watchObservedRunningTime="2026-11-03 00:28:20.205162575 +0000 UTC m=+1383.084746618"
Nov 03 00:28:26.113657 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:26.113623    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4"
Nov 03 00:28:26.280751 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:26.280715    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db"]
Nov 03 00:28:26.281001 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:26.280973    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db" podUID="8660d374-3808-4c57-8094-32e6cdc1d73d" containerName="payload-pre-processing" containerID="cri-o://e00cb6f9a92fa2b0ec8a259c2fb52528163ec0e81f9651f2584b177abf746573" gracePeriod=30
Nov 03 00:28:26.655346 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:26.654982    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db"
Nov 03 00:28:26.725124 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:26.723136    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8660d374-3808-4c57-8094-32e6cdc1d73d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8660d374-3808-4c57-8094-32e6cdc1d73d-plugins-config-volume\") pod \"8660d374-3808-4c57-8094-32e6cdc1d73d\" (UID: \"8660d374-3808-4c57-8094-32e6cdc1d73d\") "
Nov 03 00:28:26.725124 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:26.723358    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8660d374-3808-4c57-8094-32e6cdc1d73d-kube-api-access-jrtdw\" (UniqueName: \"kubernetes.io/projected/8660d374-3808-4c57-8094-32e6cdc1d73d-kube-api-access-jrtdw\") pod \"8660d374-3808-4c57-8094-32e6cdc1d73d\" (UID: \"8660d374-3808-4c57-8094-32e6cdc1d73d\") "
Nov 03 00:28:26.725124 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:26.723602    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8660d374-3808-4c57-8094-32e6cdc1d73d-plugins-config-volume" pod "8660d374-3808-4c57-8094-32e6cdc1d73d" (UID: "8660d374-3808-4c57-8094-32e6cdc1d73d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:28:26.725124 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:26.723978    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8660d374-3808-4c57-8094-32e6cdc1d73d-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:26.728388 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:26.728358    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8660d374-3808-4c57-8094-32e6cdc1d73d-kube-api-access-jrtdw" pod "8660d374-3808-4c57-8094-32e6cdc1d73d" (UID: "8660d374-3808-4c57-8094-32e6cdc1d73d"). InnerVolumeSpecName "kube-api-access-jrtdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:26.825749 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:26.825716    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrtdw\" (UniqueName: \"kubernetes.io/projected/8660d374-3808-4c57-8094-32e6cdc1d73d-kube-api-access-jrtdw\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:27.167449 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:27.167412    5012 generic.go:356] "Generic (PLEG): container finished" podID="8660d374-3808-4c57-8094-32e6cdc1d73d" containerID="e00cb6f9a92fa2b0ec8a259c2fb52528163ec0e81f9651f2584b177abf746573" exitCode=0
Nov 03 00:28:27.167851 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:27.167465    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db"
Nov 03 00:28:27.167851 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:27.167502    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db" event={"ID":"8660d374-3808-4c57-8094-32e6cdc1d73d","Type":"ContainerDied","Data":"e00cb6f9a92fa2b0ec8a259c2fb52528163ec0e81f9651f2584b177abf746573"}
Nov 03 00:28:27.167851 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:27.167560    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db" event={"ID":"8660d374-3808-4c57-8094-32e6cdc1d73d","Type":"ContainerDied","Data":"1d01bb2f9b3b7b0486097b81b9c40d4bc8edf60fdaba93d86f70f5dedf18f073"}
Nov 03 00:28:27.167851 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:27.167582    5012 scope.go:122] "RemoveContainer" containerID="e00cb6f9a92fa2b0ec8a259c2fb52528163ec0e81f9651f2584b177abf746573"
Nov 03 00:28:27.182307 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:27.182281    5012 scope.go:122] "RemoveContainer" containerID="e00cb6f9a92fa2b0ec8a259c2fb52528163ec0e81f9651f2584b177abf746573"
Nov 03 00:28:27.182692 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:28:27.182667    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e00cb6f9a92fa2b0ec8a259c2fb52528163ec0e81f9651f2584b177abf746573\": container with ID starting with e00cb6f9a92fa2b0ec8a259c2fb52528163ec0e81f9651f2584b177abf746573 not found: ID does not exist" containerID="e00cb6f9a92fa2b0ec8a259c2fb52528163ec0e81f9651f2584b177abf746573"
Nov 03 00:28:27.182779 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:27.182703    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e00cb6f9a92fa2b0ec8a259c2fb52528163ec0e81f9651f2584b177abf746573"} err="failed to get container status \"e00cb6f9a92fa2b0ec8a259c2fb52528163ec0e81f9651f2584b177abf746573\": rpc error: code = NotFound desc = could not find container \"e00cb6f9a92fa2b0ec8a259c2fb52528163ec0e81f9651f2584b177abf746573\": container with ID starting with e00cb6f9a92fa2b0ec8a259c2fb52528163ec0e81f9651f2584b177abf746573 not found: ID does not exist"
Nov 03 00:28:27.215365 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:27.215339    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db"]
Nov 03 00:28:27.219290 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:27.219254    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41e815-585ff4f9c5-mk6db"]
Nov 03 00:28:27.244535 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:27.244505    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8660d374-3808-4c57-8094-32e6cdc1d73d" path="/var/lib/kubelet/pods/8660d374-3808-4c57-8094-32e6cdc1d73d/volumes"
Nov 03 00:28:28.398799 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:28.398722    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f"]
Nov 03 00:28:28.399262 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:28.398975    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f" podUID="62381bac-7d10-4757-8d03-51b0ffe16a90" containerName="maas-api" containerID="cri-o://63748960e8e3ea90f82c3b136383db47399a4845b366ce101d43d6f43db7c538" gracePeriod=30
Nov 03 00:28:28.700620 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:28.700591    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f"
Nov 03 00:28:28.849634 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:28.849606    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62381bac-7d10-4757-8d03-51b0ffe16a90-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62381bac-7d10-4757-8d03-51b0ffe16a90-openshift-service-ca\") pod \"62381bac-7d10-4757-8d03-51b0ffe16a90\" (UID: \"62381bac-7d10-4757-8d03-51b0ffe16a90\") "
Nov 03 00:28:28.849788 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:28.849700    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62381bac-7d10-4757-8d03-51b0ffe16a90-metrics-certs\" (UniqueName: \"kubernetes.io/secret/62381bac-7d10-4757-8d03-51b0ffe16a90-metrics-certs\") pod \"62381bac-7d10-4757-8d03-51b0ffe16a90\" (UID: \"62381bac-7d10-4757-8d03-51b0ffe16a90\") "
Nov 03 00:28:28.849859 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:28.849815    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62381bac-7d10-4757-8d03-51b0ffe16a90-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62381bac-7d10-4757-8d03-51b0ffe16a90-maas-api-tls\") pod \"62381bac-7d10-4757-8d03-51b0ffe16a90\" (UID: \"62381bac-7d10-4757-8d03-51b0ffe16a90\") "
Nov 03 00:28:28.849924 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:28.849869    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62381bac-7d10-4757-8d03-51b0ffe16a90-kube-api-access-shbfs\" (UniqueName: \"kubernetes.io/projected/62381bac-7d10-4757-8d03-51b0ffe16a90-kube-api-access-shbfs\") pod \"62381bac-7d10-4757-8d03-51b0ffe16a90\" (UID: \"62381bac-7d10-4757-8d03-51b0ffe16a90\") "
Nov 03 00:28:28.850050 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:28.850028    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62381bac-7d10-4757-8d03-51b0ffe16a90-openshift-service-ca" pod "62381bac-7d10-4757-8d03-51b0ffe16a90" (UID: "62381bac-7d10-4757-8d03-51b0ffe16a90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:28:28.850483 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:28.850458    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62381bac-7d10-4757-8d03-51b0ffe16a90-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:28.851959 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:28.851924    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62381bac-7d10-4757-8d03-51b0ffe16a90-kube-api-access-shbfs" pod "62381bac-7d10-4757-8d03-51b0ffe16a90" (UID: "62381bac-7d10-4757-8d03-51b0ffe16a90"). InnerVolumeSpecName "kube-api-access-shbfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:28.852062 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:28.851959    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62381bac-7d10-4757-8d03-51b0ffe16a90-maas-api-tls" pod "62381bac-7d10-4757-8d03-51b0ffe16a90" (UID: "62381bac-7d10-4757-8d03-51b0ffe16a90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:28:28.852123 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:28.852086    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62381bac-7d10-4757-8d03-51b0ffe16a90-metrics-certs" pod "62381bac-7d10-4757-8d03-51b0ffe16a90" (UID: "62381bac-7d10-4757-8d03-51b0ffe16a90"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:28:28.951689 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:28.951603    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62381bac-7d10-4757-8d03-51b0ffe16a90-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:28.951689 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:28.951636    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62381bac-7d10-4757-8d03-51b0ffe16a90-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:28.951689 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:28.951646    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shbfs\" (UniqueName: \"kubernetes.io/projected/62381bac-7d10-4757-8d03-51b0ffe16a90-kube-api-access-shbfs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:29.186376 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:29.186336    5012 generic.go:356] "Generic (PLEG): container finished" podID="62381bac-7d10-4757-8d03-51b0ffe16a90" containerID="63748960e8e3ea90f82c3b136383db47399a4845b366ce101d43d6f43db7c538" exitCode=0
Nov 03 00:28:29.186666 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:29.186384    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f"
Nov 03 00:28:29.186666 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:29.186397    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f" event={"ID":"62381bac-7d10-4757-8d03-51b0ffe16a90","Type":"ContainerDied","Data":"63748960e8e3ea90f82c3b136383db47399a4845b366ce101d43d6f43db7c538"}
Nov 03 00:28:29.186666 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:29.186436    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f" event={"ID":"62381bac-7d10-4757-8d03-51b0ffe16a90","Type":"ContainerDied","Data":"5a52ed4bc97ba7ff6e1eaa183291db0a135156fd815e5010ac5efcf893a5b3ea"}
Nov 03 00:28:29.186666 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:29.186465    5012 scope.go:122] "RemoveContainer" containerID="63748960e8e3ea90f82c3b136383db47399a4845b366ce101d43d6f43db7c538"
Nov 03 00:28:29.199729 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:29.199707    5012 scope.go:122] "RemoveContainer" containerID="63748960e8e3ea90f82c3b136383db47399a4845b366ce101d43d6f43db7c538"
Nov 03 00:28:29.200034 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:28:29.200007    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63748960e8e3ea90f82c3b136383db47399a4845b366ce101d43d6f43db7c538\": container with ID starting with 63748960e8e3ea90f82c3b136383db47399a4845b366ce101d43d6f43db7c538 not found: ID does not exist" containerID="63748960e8e3ea90f82c3b136383db47399a4845b366ce101d43d6f43db7c538"
Nov 03 00:28:29.200108 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:29.200041    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63748960e8e3ea90f82c3b136383db47399a4845b366ce101d43d6f43db7c538"} err="failed to get container status \"63748960e8e3ea90f82c3b136383db47399a4845b366ce101d43d6f43db7c538\": rpc error: code = NotFound desc = could not find container \"63748960e8e3ea90f82c3b136383db47399a4845b366ce101d43d6f43db7c538\": container with ID starting with 63748960e8e3ea90f82c3b136383db47399a4845b366ce101d43d6f43db7c538 not found: ID does not exist"
Nov 03 00:28:29.235559 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:29.235509    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f"]
Nov 03 00:28:29.250894 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:29.250866    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1d29dd-6df9f647b4-b924f"]
Nov 03 00:28:31.110221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:31.110193    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9"
Nov 03 00:28:31.110788 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:31.110261    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj"
Nov 03 00:28:31.246081 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:31.246049    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62381bac-7d10-4757-8d03-51b0ffe16a90" path="/var/lib/kubelet/pods/62381bac-7d10-4757-8d03-51b0ffe16a90/volumes"
Nov 03 00:28:36.030283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.030245    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9"]
Nov 03 00:28:36.068300 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.068252    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk"]
Nov 03 00:28:36.126924 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.126867    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"]
Nov 03 00:28:36.128301 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.128067    5012 cpu_manager.go:414] "containerMap: removing container" podUID="62381bac-7d10-4757-8d03-51b0ffe16a90" containerName="maas-api"
Nov 03 00:28:36.128301 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.128092    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62381bac-7d10-4757-8d03-51b0ffe16a90" containerName="maas-api"
Nov 03 00:28:36.128301 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.128131    5012 cpu_manager.go:414] "containerMap: removing container" podUID="8660d374-3808-4c57-8094-32e6cdc1d73d" containerName="payload-pre-processing"
Nov 03 00:28:36.128301 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.128139    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8660d374-3808-4c57-8094-32e6cdc1d73d" containerName="payload-pre-processing"
Nov 03 00:28:36.128301 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.128306    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="8660d374-3808-4c57-8094-32e6cdc1d73d" containerName="payload-pre-processing"
Nov 03 00:28:36.128642 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.128329    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="62381bac-7d10-4757-8d03-51b0ffe16a90" containerName="maas-api"
Nov 03 00:28:36.134100 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.133581    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.144414 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.144380    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"]
Nov 03 00:28:36.147202 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.147178    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-846492-openshift-default-dockercfg-42dnn\""
Nov 03 00:28:36.278281 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.278241    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-envoy\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.278592 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.278358    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-workload-socket\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.278592 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.278402    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-data\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.278729 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.278681    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-workload-certs\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.278835 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.278810    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k58h9\" (UniqueName: \"kubernetes.io/projected/5e8e36a8-3445-41ab-a889-c529c3e707c9-kube-api-access-k58h9\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.278905 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.278849    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-credential-socket\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.279035 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.278994    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8e36a8-3445-41ab-a889-c529c3e707c9-istiod-ca-cert\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.279135 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.279114    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-podinfo\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.279225 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.279202    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-token\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.380099 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.380062    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k58h9\" (UniqueName: \"kubernetes.io/projected/5e8e36a8-3445-41ab-a889-c529c3e707c9-kube-api-access-k58h9\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.380277 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.380108    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-credential-socket\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.380277 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.380174    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8e36a8-3445-41ab-a889-c529c3e707c9-istiod-ca-cert\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.380277 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.380247    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-podinfo\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.380465 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.380285    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-token\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.380465 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.380350    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-envoy\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.380465 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.380418    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-workload-socket\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.380465 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.380457    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-data\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.380678 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.380519    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-workload-certs\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.380678 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.380640    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-credential-socket\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.381110 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.381084    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8e36a8-3445-41ab-a889-c529c3e707c9-istiod-ca-cert\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.381218 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.381113    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-data\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.381284 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.381224    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-workload-certs\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.381477 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.381454    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-workload-socket\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.383200 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.383177    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-podinfo\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.383496 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.383475    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-envoy\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.437558 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.436801    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-token\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.445290 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.445256    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k58h9\" (UniqueName: \"kubernetes.io/projected/5e8e36a8-3445-41ab-a889-c529c3e707c9-kube-api-access-k58h9\") pod \"e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") " pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:36.463337 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:36.463303    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:37.165029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:37.165003    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"]
Nov 03 00:28:37.165478 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:28:37.165454    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8e36a8_3445_41ab_a889_c529c3e707c9.slice/crio-0d503760a94923602c5951f41b1e0f9ff18ce21a32609c44bf7712c3493a2131 WatchSource:0}: Error finding container 0d503760a94923602c5951f41b1e0f9ff18ce21a32609c44bf7712c3493a2131: Status 404 returned error can't find the container with id 0d503760a94923602c5951f41b1e0f9ff18ce21a32609c44bf7712c3493a2131
Nov 03 00:28:37.168570 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:37.168536    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:28:37.168670 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:37.168602    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:28:37.168670 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:37.168636    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:28:37.257324 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:37.257297    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz" event={"ID":"5e8e36a8-3445-41ab-a889-c529c3e707c9","Type":"ContainerStarted","Data":"0d503760a94923602c5951f41b1e0f9ff18ce21a32609c44bf7712c3493a2131"}
Nov 03 00:28:37.365283 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:28:37.365247    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:28:38.267079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:38.267039    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz" event={"ID":"5e8e36a8-3445-41ab-a889-c529c3e707c9","Type":"ContainerStarted","Data":"cd96a43eed3105271ace6242ac2fc21431a985189829b6a578bf4b6d381089ab"}
Nov 03 00:28:38.343965 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:38.342373    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz" podStartSLOduration=2.342357868 podStartE2EDuration="2.342357868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:28:38.324450658 +0000 UTC m=+1401.204034701" watchObservedRunningTime="2026-11-03 00:28:38.342357868 +0000 UTC m=+1401.221941992"
Nov 03 00:28:38.465237 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:38.464982    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:38.477943 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:38.474524    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:39.284041 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:39.283965    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:39.286278 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:39.285961    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:28:41.031264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:41.031225    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9" podUID="911edb88-1e60-40b0-843e-7086beabbc9b" containerName="payload-processing" containerID="cri-o://6f0e9929b9acd16306e01b4c07dc72becf81e8d94c2cfa7c12ae071f38543193" gracePeriod=25
Nov 03 00:28:41.069102 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:41.069046    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk" podUID="6ab5fda5-8837-454e-9554-941f04251283" containerName="payload-pre-processing" containerID="cri-o://3d00e032103f2ced513f798e31fbb0951eedcffc29ab403243eb762561642b0c" gracePeriod=25
Nov 03 00:28:41.330966 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:41.330873    5012 generic.go:356] "Generic (PLEG): container finished" podID="6ab5fda5-8837-454e-9554-941f04251283" containerID="3d00e032103f2ced513f798e31fbb0951eedcffc29ab403243eb762561642b0c" exitCode=0
Nov 03 00:28:41.331118 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:41.330964    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk" event={"ID":"6ab5fda5-8837-454e-9554-941f04251283","Type":"ContainerDied","Data":"3d00e032103f2ced513f798e31fbb0951eedcffc29ab403243eb762561642b0c"}
Nov 03 00:28:41.335453 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:41.335394    5012 generic.go:356] "Generic (PLEG): container finished" podID="911edb88-1e60-40b0-843e-7086beabbc9b" containerID="6f0e9929b9acd16306e01b4c07dc72becf81e8d94c2cfa7c12ae071f38543193" exitCode=0
Nov 03 00:28:41.335592 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:41.335454    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9" event={"ID":"911edb88-1e60-40b0-843e-7086beabbc9b","Type":"ContainerDied","Data":"6f0e9929b9acd16306e01b4c07dc72becf81e8d94c2cfa7c12ae071f38543193"}
Nov 03 00:28:41.448047 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:41.448017    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9"
Nov 03 00:28:41.505940 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:41.496071    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk"
Nov 03 00:28:41.544998 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:41.544963    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ab5fda5-8837-454e-9554-941f04251283-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab5fda5-8837-454e-9554-941f04251283-plugins-config-volume\") pod \"6ab5fda5-8837-454e-9554-941f04251283\" (UID: \"6ab5fda5-8837-454e-9554-941f04251283\") "
Nov 03 00:28:41.545181 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:41.545048    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/911edb88-1e60-40b0-843e-7086beabbc9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911edb88-1e60-40b0-843e-7086beabbc9b-plugins-config-volume\") pod \"911edb88-1e60-40b0-843e-7086beabbc9b\" (UID: \"911edb88-1e60-40b0-843e-7086beabbc9b\") "
Nov 03 00:28:41.545181 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:41.545150    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab5fda5-8837-454e-9554-941f04251283-kube-api-access-m8b4s\" (UniqueName: \"kubernetes.io/projected/6ab5fda5-8837-454e-9554-941f04251283-kube-api-access-m8b4s\") pod \"6ab5fda5-8837-454e-9554-941f04251283\" (UID: \"6ab5fda5-8837-454e-9554-941f04251283\") "
Nov 03 00:28:41.545326 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:41.545303    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/911edb88-1e60-40b0-843e-7086beabbc9b-kube-api-access-59ds2\" (UniqueName: \"kubernetes.io/projected/911edb88-1e60-40b0-843e-7086beabbc9b-kube-api-access-59ds2\") pod \"911edb88-1e60-40b0-843e-7086beabbc9b\" (UID: \"911edb88-1e60-40b0-843e-7086beabbc9b\") "
Nov 03 00:28:41.545415 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:41.545385    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab5fda5-8837-454e-9554-941f04251283-plugins-config-volume" pod "6ab5fda5-8837-454e-9554-941f04251283" (UID: "6ab5fda5-8837-454e-9554-941f04251283"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:28:41.545815 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:41.545790    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/911edb88-1e60-40b0-843e-7086beabbc9b-plugins-config-volume" pod "911edb88-1e60-40b0-843e-7086beabbc9b" (UID: "911edb88-1e60-40b0-843e-7086beabbc9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:28:41.546006 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:41.545975    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab5fda5-8837-454e-9554-941f04251283-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:41.546006 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:41.545999    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911edb88-1e60-40b0-843e-7086beabbc9b-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:41.548185 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:41.548156    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/911edb88-1e60-40b0-843e-7086beabbc9b-kube-api-access-59ds2" pod "911edb88-1e60-40b0-843e-7086beabbc9b" (UID: "911edb88-1e60-40b0-843e-7086beabbc9b"). InnerVolumeSpecName "kube-api-access-59ds2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:41.550488 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:41.550463    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab5fda5-8837-454e-9554-941f04251283-kube-api-access-m8b4s" pod "6ab5fda5-8837-454e-9554-941f04251283" (UID: "6ab5fda5-8837-454e-9554-941f04251283"). InnerVolumeSpecName "kube-api-access-m8b4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:41.647583 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:41.647544    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59ds2\" (UniqueName: \"kubernetes.io/projected/911edb88-1e60-40b0-843e-7086beabbc9b-kube-api-access-59ds2\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:41.647583 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:41.647584    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8b4s\" (UniqueName: \"kubernetes.io/projected/6ab5fda5-8837-454e-9554-941f04251283-kube-api-access-m8b4s\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:42.346937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.346886    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk"
Nov 03 00:28:42.347359 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.346939    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk" event={"ID":"6ab5fda5-8837-454e-9554-941f04251283","Type":"ContainerDied","Data":"2d6f147014f1248251047e49d2b7d3d03e247f068a953c9f954977795f44592c"}
Nov 03 00:28:42.347359 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.347005    5012 scope.go:122] "RemoveContainer" containerID="3d00e032103f2ced513f798e31fbb0951eedcffc29ab403243eb762561642b0c"
Nov 03 00:28:42.348939 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.348880    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9" event={"ID":"911edb88-1e60-40b0-843e-7086beabbc9b","Type":"ContainerDied","Data":"8a0693b45dfb4225a4300eb38e759d4461198e0d9ae7c61299152d9fd978818b"}
Nov 03 00:28:42.349128 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.349075    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9"
Nov 03 00:28:42.369046 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.369020    5012 scope.go:122] "RemoveContainer" containerID="6f0e9929b9acd16306e01b4c07dc72becf81e8d94c2cfa7c12ae071f38543193"
Nov 03 00:28:42.423532 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.423500    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk"]
Nov 03 00:28:42.435115 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.435081    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-5956785455-r4bfk"]
Nov 03 00:28:42.465036 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.464879    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9"]
Nov 03 00:28:42.471364 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.471334    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-6886954d5-s9nb9"]
Nov 03 00:28:42.553462 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.553430    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j"]
Nov 03 00:28:42.554128 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.554107    5012 cpu_manager.go:414] "containerMap: removing container" podUID="6ab5fda5-8837-454e-9554-941f04251283" containerName="payload-pre-processing"
Nov 03 00:28:42.554128 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.554121    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab5fda5-8837-454e-9554-941f04251283" containerName="payload-pre-processing"
Nov 03 00:28:42.554306 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.554140    5012 cpu_manager.go:414] "containerMap: removing container" podUID="911edb88-1e60-40b0-843e-7086beabbc9b" containerName="payload-processing"
Nov 03 00:28:42.554306 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.554146    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="911edb88-1e60-40b0-843e-7086beabbc9b" containerName="payload-processing"
Nov 03 00:28:42.554306 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.554286    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab5fda5-8837-454e-9554-941f04251283" containerName="payload-pre-processing"
Nov 03 00:28:42.554306 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.554298    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="911edb88-1e60-40b0-843e-7086beabbc9b" containerName="payload-processing"
Nov 03 00:28:42.559323 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.559302    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j"
Nov 03 00:28:42.574164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.574106    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-846492\""
Nov 03 00:28:42.576506 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.576482    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j"]
Nov 03 00:28:42.663610 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.663514    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b49605-cfa9-464b-a91e-ec48f74cbef8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j\" (UID: \"29b49605-cfa9-464b-a91e-ec48f74cbef8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j"
Nov 03 00:28:42.663949 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.663759    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tbzx\" (UniqueName: \"kubernetes.io/projected/29b49605-cfa9-464b-a91e-ec48f74cbef8-kube-api-access-9tbzx\") pod \"payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j\" (UID: \"29b49605-cfa9-464b-a91e-ec48f74cbef8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j"
Nov 03 00:28:42.810947 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.787696    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tbzx\" (UniqueName: \"kubernetes.io/projected/29b49605-cfa9-464b-a91e-ec48f74cbef8-kube-api-access-9tbzx\") pod \"payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j\" (UID: \"29b49605-cfa9-464b-a91e-ec48f74cbef8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j"
Nov 03 00:28:42.810947 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.787791    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b49605-cfa9-464b-a91e-ec48f74cbef8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j\" (UID: \"29b49605-cfa9-464b-a91e-ec48f74cbef8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j"
Nov 03 00:28:42.810947 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.788766    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b49605-cfa9-464b-a91e-ec48f74cbef8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j\" (UID: \"29b49605-cfa9-464b-a91e-ec48f74cbef8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j"
Nov 03 00:28:42.841795 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.841756    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tbzx\" (UniqueName: \"kubernetes.io/projected/29b49605-cfa9-464b-a91e-ec48f74cbef8-kube-api-access-9tbzx\") pod \"payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j\" (UID: \"29b49605-cfa9-464b-a91e-ec48f74cbef8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j"
Nov 03 00:28:42.880549 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:42.880515    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j"
Nov 03 00:28:43.236631 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:43.235103    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j"]
Nov 03 00:28:43.251088 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:43.251060    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ab5fda5-8837-454e-9554-941f04251283" path="/var/lib/kubelet/pods/6ab5fda5-8837-454e-9554-941f04251283/volumes"
Nov 03 00:28:43.251632 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:43.251613    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="911edb88-1e60-40b0-843e-7086beabbc9b" path="/var/lib/kubelet/pods/911edb88-1e60-40b0-843e-7086beabbc9b/volumes"
Nov 03 00:28:43.358814 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:43.358777    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j" event={"ID":"29b49605-cfa9-464b-a91e-ec48f74cbef8","Type":"ContainerStarted","Data":"c098c87fdabd6f2efb3a7a6b240f35ccdf36076c14dba6f75f8f88afe783f5ee"}
Nov 03 00:28:43.359229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:43.358847    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j" event={"ID":"29b49605-cfa9-464b-a91e-ec48f74cbef8","Type":"ContainerStarted","Data":"c0076a0f88894185b289fbbd78e9d59769b043fad181fb42654480d012476a0c"}
Nov 03 00:28:43.359229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:43.358884    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j"
Nov 03 00:28:43.396879 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:43.396829    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j" podStartSLOduration=1.396816018 podStartE2EDuration="1.396816018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:28:43.393462162 +0000 UTC m=+1406.273046204" watchObservedRunningTime="2026-11-03 00:28:43.396816018 +0000 UTC m=+1406.276400075"
Nov 03 00:28:44.372197 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:44.372165    5012 generic.go:356] "Generic (PLEG): container finished" podID="b4f51e9c-5f2c-41df-a26f-4978e47bf980" containerID="81e9a006c55992de198eb569c21ec23b96c9a96cfc3c317c947c754e27f334d4" exitCode=0
Nov 03 00:28:44.372619 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:44.372234    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h" event={"ID":"b4f51e9c-5f2c-41df-a26f-4978e47bf980","Type":"ContainerDied","Data":"81e9a006c55992de198eb569c21ec23b96c9a96cfc3c317c947c754e27f334d4"}
Nov 03 00:28:44.372815 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:44.372792    5012 scope.go:122] "RemoveContainer" containerID="81e9a006c55992de198eb569c21ec23b96c9a96cfc3c317c947c754e27f334d4"
Nov 03 00:28:44.374260 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:44.374222    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:28:44.374354 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:44.374293    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:28:44.374354 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:44.374326    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:28:44.656029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:44.655991    5012 patch_prober.go:29] interesting pod/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:28:44.656247 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:44.656068    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms" podUID="ba93722b-9169-4628-9f61-baaef18a68a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:28:45.119178 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:45.119144    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh"]
Nov 03 00:28:45.152179 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:45.152149    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j"]
Nov 03 00:28:45.384105 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:45.384008    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h" event={"ID":"b4f51e9c-5f2c-41df-a26f-4978e47bf980","Type":"ContainerStarted","Data":"ac896fba81a2ef1ae9f6c3a3e8af8e0048916bc7ee3b6bbf9a023fba745f2765"}
Nov 03 00:28:45.762985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:45.762481    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:28:45.768934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:45.767056    5012 patch_prober.go:29] interesting pod/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 03 00:28:45.768934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:45.767125    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h" podUID="b4f51e9c-5f2c-41df-a26f-4978e47bf980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 03 00:28:46.761960 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:46.761926    5012 patch_prober.go:29] interesting pod/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 03 00:28:46.762364 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:46.761981    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h" podUID="b4f51e9c-5f2c-41df-a26f-4978e47bf980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 03 00:28:47.721065 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:47.720993    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr"]
Nov 03 00:28:47.721255 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:47.721177    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr" podUID="3dc8852e-27d6-4295-8580-cb92fcea2eaf" containerName="payload-processing" containerID="cri-o://c5535c7ab052580fad527fff2a1cfe845953291a879e14dd7129fc97192e9ff9" gracePeriod=30
Nov 03 00:28:47.751009 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:47.750984    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd"]
Nov 03 00:28:47.751212 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:47.751191    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd" podUID="860bdb4a-93f0-45de-b939-8471e7b7aecb" containerName="payload-pre-processing" containerID="cri-o://b6471b3426efd6cdfc9fc5c62242a5db391f3f6f369279de617390be151398fb" gracePeriod=30
Nov 03 00:28:47.778115 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:47.778081    5012 patch_prober.go:29] interesting pod/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 03 00:28:47.778615 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:47.778134    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h" podUID="b4f51e9c-5f2c-41df-a26f-4978e47bf980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 03 00:28:48.162969 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.162904    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd"
Nov 03 00:28:48.270395 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.270360    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/860bdb4a-93f0-45de-b939-8471e7b7aecb-kube-api-access-95lgj\" (UniqueName: \"kubernetes.io/projected/860bdb4a-93f0-45de-b939-8471e7b7aecb-kube-api-access-95lgj\") pod \"860bdb4a-93f0-45de-b939-8471e7b7aecb\" (UID: \"860bdb4a-93f0-45de-b939-8471e7b7aecb\") "
Nov 03 00:28:48.271226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.271196    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/860bdb4a-93f0-45de-b939-8471e7b7aecb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/860bdb4a-93f0-45de-b939-8471e7b7aecb-plugins-config-volume\") pod \"860bdb4a-93f0-45de-b939-8471e7b7aecb\" (UID: \"860bdb4a-93f0-45de-b939-8471e7b7aecb\") "
Nov 03 00:28:48.271690 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.271661    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/860bdb4a-93f0-45de-b939-8471e7b7aecb-plugins-config-volume" pod "860bdb4a-93f0-45de-b939-8471e7b7aecb" (UID: "860bdb4a-93f0-45de-b939-8471e7b7aecb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:28:48.273478 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.273217    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860bdb4a-93f0-45de-b939-8471e7b7aecb-kube-api-access-95lgj" pod "860bdb4a-93f0-45de-b939-8471e7b7aecb" (UID: "860bdb4a-93f0-45de-b939-8471e7b7aecb"). InnerVolumeSpecName "kube-api-access-95lgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:48.274058 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.274028    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/860bdb4a-93f0-45de-b939-8471e7b7aecb-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:48.275101 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.275083    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95lgj\" (UniqueName: \"kubernetes.io/projected/860bdb4a-93f0-45de-b939-8471e7b7aecb-kube-api-access-95lgj\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:48.295462 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.295137    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr"
Nov 03 00:28:48.376037 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.375958    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dc8852e-27d6-4295-8580-cb92fcea2eaf-kube-api-access-xqr6b\" (UniqueName: \"kubernetes.io/projected/3dc8852e-27d6-4295-8580-cb92fcea2eaf-kube-api-access-xqr6b\") pod \"3dc8852e-27d6-4295-8580-cb92fcea2eaf\" (UID: \"3dc8852e-27d6-4295-8580-cb92fcea2eaf\") "
Nov 03 00:28:48.376037 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.376032    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dc8852e-27d6-4295-8580-cb92fcea2eaf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc8852e-27d6-4295-8580-cb92fcea2eaf-plugins-config-volume\") pod \"3dc8852e-27d6-4295-8580-cb92fcea2eaf\" (UID: \"3dc8852e-27d6-4295-8580-cb92fcea2eaf\") "
Nov 03 00:28:48.376500 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.376469    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dc8852e-27d6-4295-8580-cb92fcea2eaf-plugins-config-volume" pod "3dc8852e-27d6-4295-8580-cb92fcea2eaf" (UID: "3dc8852e-27d6-4295-8580-cb92fcea2eaf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:28:48.376740 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.376718    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc8852e-27d6-4295-8580-cb92fcea2eaf-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:48.378189 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.378158    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dc8852e-27d6-4295-8580-cb92fcea2eaf-kube-api-access-xqr6b" pod "3dc8852e-27d6-4295-8580-cb92fcea2eaf" (UID: "3dc8852e-27d6-4295-8580-cb92fcea2eaf"). InnerVolumeSpecName "kube-api-access-xqr6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:48.436238 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.436208    5012 generic.go:356] "Generic (PLEG): container finished" podID="ba93722b-9169-4628-9f61-baaef18a68a7" containerID="f4878d8508f1d06d42d201a2f1dcd80e113ab7824553830502fc5ce404ce1390" exitCode=0
Nov 03 00:28:48.436418 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.436286    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms" event={"ID":"ba93722b-9169-4628-9f61-baaef18a68a7","Type":"ContainerDied","Data":"f4878d8508f1d06d42d201a2f1dcd80e113ab7824553830502fc5ce404ce1390"}
Nov 03 00:28:48.436778 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.436761    5012 scope.go:122] "RemoveContainer" containerID="f4878d8508f1d06d42d201a2f1dcd80e113ab7824553830502fc5ce404ce1390"
Nov 03 00:28:48.437728 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.437708    5012 generic.go:356] "Generic (PLEG): container finished" podID="860bdb4a-93f0-45de-b939-8471e7b7aecb" containerID="b6471b3426efd6cdfc9fc5c62242a5db391f3f6f369279de617390be151398fb" exitCode=0
Nov 03 00:28:48.437827 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.437749    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd"
Nov 03 00:28:48.437827 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.437792    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd" event={"ID":"860bdb4a-93f0-45de-b939-8471e7b7aecb","Type":"ContainerDied","Data":"b6471b3426efd6cdfc9fc5c62242a5db391f3f6f369279de617390be151398fb"}
Nov 03 00:28:48.437951 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.437838    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd" event={"ID":"860bdb4a-93f0-45de-b939-8471e7b7aecb","Type":"ContainerDied","Data":"ee85960016771e2544631d199ebca89f5d6bca0d0ddfd27e089f6fed5b35f112"}
Nov 03 00:28:48.437951 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.437869    5012 scope.go:122] "RemoveContainer" containerID="b6471b3426efd6cdfc9fc5c62242a5db391f3f6f369279de617390be151398fb"
Nov 03 00:28:48.438381 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.438351    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:28:48.438477 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.438461    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:28:48.438551 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.438496    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:28:48.439675 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.439653    5012 generic.go:356] "Generic (PLEG): container finished" podID="3dc8852e-27d6-4295-8580-cb92fcea2eaf" containerID="c5535c7ab052580fad527fff2a1cfe845953291a879e14dd7129fc97192e9ff9" exitCode=0
Nov 03 00:28:48.439752 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.439702    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr"
Nov 03 00:28:48.439838 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.439818    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr" event={"ID":"3dc8852e-27d6-4295-8580-cb92fcea2eaf","Type":"ContainerDied","Data":"c5535c7ab052580fad527fff2a1cfe845953291a879e14dd7129fc97192e9ff9"}
Nov 03 00:28:48.439995 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.439866    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr" event={"ID":"3dc8852e-27d6-4295-8580-cb92fcea2eaf","Type":"ContainerDied","Data":"9d03c72d451e71449969a6978ba420892d8f33ddffb5089b962e0746a470cbca"}
Nov 03 00:28:48.471407 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.471388    5012 scope.go:122] "RemoveContainer" containerID="b6471b3426efd6cdfc9fc5c62242a5db391f3f6f369279de617390be151398fb"
Nov 03 00:28:48.471692 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:28:48.471648    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6471b3426efd6cdfc9fc5c62242a5db391f3f6f369279de617390be151398fb\": container with ID starting with b6471b3426efd6cdfc9fc5c62242a5db391f3f6f369279de617390be151398fb not found: ID does not exist" containerID="b6471b3426efd6cdfc9fc5c62242a5db391f3f6f369279de617390be151398fb"
Nov 03 00:28:48.471810 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.471697    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6471b3426efd6cdfc9fc5c62242a5db391f3f6f369279de617390be151398fb"} err="failed to get container status \"b6471b3426efd6cdfc9fc5c62242a5db391f3f6f369279de617390be151398fb\": rpc error: code = NotFound desc = could not find container \"b6471b3426efd6cdfc9fc5c62242a5db391f3f6f369279de617390be151398fb\": container with ID starting with b6471b3426efd6cdfc9fc5c62242a5db391f3f6f369279de617390be151398fb not found: ID does not exist"
Nov 03 00:28:48.471810 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.471720    5012 scope.go:122] "RemoveContainer" containerID="c5535c7ab052580fad527fff2a1cfe845953291a879e14dd7129fc97192e9ff9"
Nov 03 00:28:48.477939 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.477879    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqr6b\" (UniqueName: \"kubernetes.io/projected/3dc8852e-27d6-4295-8580-cb92fcea2eaf-kube-api-access-xqr6b\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:48.485165 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.485145    5012 scope.go:122] "RemoveContainer" containerID="c5535c7ab052580fad527fff2a1cfe845953291a879e14dd7129fc97192e9ff9"
Nov 03 00:28:48.490960 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:28:48.485905    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5535c7ab052580fad527fff2a1cfe845953291a879e14dd7129fc97192e9ff9\": container with ID starting with c5535c7ab052580fad527fff2a1cfe845953291a879e14dd7129fc97192e9ff9 not found: ID does not exist" containerID="c5535c7ab052580fad527fff2a1cfe845953291a879e14dd7129fc97192e9ff9"
Nov 03 00:28:48.490960 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.486000    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5535c7ab052580fad527fff2a1cfe845953291a879e14dd7129fc97192e9ff9"} err="failed to get container status \"c5535c7ab052580fad527fff2a1cfe845953291a879e14dd7129fc97192e9ff9\": rpc error: code = NotFound desc = could not find container \"c5535c7ab052580fad527fff2a1cfe845953291a879e14dd7129fc97192e9ff9\": container with ID starting with c5535c7ab052580fad527fff2a1cfe845953291a879e14dd7129fc97192e9ff9 not found: ID does not exist"
Nov 03 00:28:48.521935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.521762    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd"]
Nov 03 00:28:48.527110 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.527090    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dda6df-86545c69d7-9pjsd"]
Nov 03 00:28:48.556277 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.556252    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr"]
Nov 03 00:28:48.560332 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.560311    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dda6df-6b54c6dc5d-5n4dr"]
Nov 03 00:28:48.761751 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.761680    5012 patch_prober.go:29] interesting pod/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 03 00:28:48.761751 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:48.761727    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h" podUID="b4f51e9c-5f2c-41df-a26f-4978e47bf980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 03 00:28:49.246192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:49.246161    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dc8852e-27d6-4295-8580-cb92fcea2eaf" path="/var/lib/kubelet/pods/3dc8852e-27d6-4295-8580-cb92fcea2eaf/volumes"
Nov 03 00:28:49.246628 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:49.246613    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="860bdb4a-93f0-45de-b939-8471e7b7aecb" path="/var/lib/kubelet/pods/860bdb4a-93f0-45de-b939-8471e7b7aecb/volumes"
Nov 03 00:28:49.450923 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:49.450886    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms" event={"ID":"ba93722b-9169-4628-9f61-baaef18a68a7","Type":"ContainerStarted","Data":"5c497e9f8aff8800645c483955230201067624b6ddc3e390faa03d406767d944"}
Nov 03 00:28:49.762131 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:49.762096    5012 patch_prober.go:29] interesting pod/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 03 00:28:49.762309 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:49.762150    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h" podUID="b4f51e9c-5f2c-41df-a26f-4978e47bf980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 03 00:28:50.122041 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.121975    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh" podUID="61049872-551b-4d49-bf01-ed17bbf2353e" containerName="payload-processing" containerID="cri-o://fb83a95b13e1a19e932e7c63e0393d100a977aa4e95c2be01ff8280191141879" gracePeriod=25
Nov 03 00:28:50.152605 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.152565    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j" podUID="1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef" containerName="payload-pre-processing" containerID="cri-o://aca9b9149b2893bbd9f615f77ffcc4682d230b354926d0d0ded7e53de305d880" gracePeriod=25
Nov 03 00:28:50.207932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.207253    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:28:50.207932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.207370    5012 patch_prober.go:29] interesting pod/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 03 00:28:50.207932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.207425    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms" podUID="ba93722b-9169-4628-9f61-baaef18a68a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 03 00:28:50.468950 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.468734    5012 generic.go:356] "Generic (PLEG): container finished" podID="61049872-551b-4d49-bf01-ed17bbf2353e" containerID="fb83a95b13e1a19e932e7c63e0393d100a977aa4e95c2be01ff8280191141879" exitCode=0
Nov 03 00:28:50.468950 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.468806    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh" event={"ID":"61049872-551b-4d49-bf01-ed17bbf2353e","Type":"ContainerDied","Data":"fb83a95b13e1a19e932e7c63e0393d100a977aa4e95c2be01ff8280191141879"}
Nov 03 00:28:50.471322 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.471277    5012 generic.go:356] "Generic (PLEG): container finished" podID="1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef" containerID="aca9b9149b2893bbd9f615f77ffcc4682d230b354926d0d0ded7e53de305d880" exitCode=0
Nov 03 00:28:50.472455 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.472196    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j" event={"ID":"1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef","Type":"ContainerDied","Data":"aca9b9149b2893bbd9f615f77ffcc4682d230b354926d0d0ded7e53de305d880"}
Nov 03 00:28:50.569025 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.568983    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j"
Nov 03 00:28:50.635174 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.635098    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh"
Nov 03 00:28:50.656731 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.656696    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef-kube-api-access-2bj9v\" (UniqueName: \"kubernetes.io/projected/1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef-kube-api-access-2bj9v\") pod \"1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef\" (UID: \"1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef\") "
Nov 03 00:28:50.656893 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.656830    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef-plugins-config-volume\") pod \"1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef\" (UID: \"1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef\") "
Nov 03 00:28:50.658383 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.658323    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef-plugins-config-volume" pod "1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef" (UID: "1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:28:50.661365 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.661336    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef-kube-api-access-2bj9v" pod "1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef" (UID: "1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef"). InnerVolumeSpecName "kube-api-access-2bj9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:50.757995 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.757966    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61049872-551b-4d49-bf01-ed17bbf2353e-kube-api-access-lmwhr\" (UniqueName: \"kubernetes.io/projected/61049872-551b-4d49-bf01-ed17bbf2353e-kube-api-access-lmwhr\") pod \"61049872-551b-4d49-bf01-ed17bbf2353e\" (UID: \"61049872-551b-4d49-bf01-ed17bbf2353e\") "
Nov 03 00:28:50.758171 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.758155    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61049872-551b-4d49-bf01-ed17bbf2353e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61049872-551b-4d49-bf01-ed17bbf2353e-plugins-config-volume\") pod \"61049872-551b-4d49-bf01-ed17bbf2353e\" (UID: \"61049872-551b-4d49-bf01-ed17bbf2353e\") "
Nov 03 00:28:50.758535 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.758508    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61049872-551b-4d49-bf01-ed17bbf2353e-plugins-config-volume" pod "61049872-551b-4d49-bf01-ed17bbf2353e" (UID: "61049872-551b-4d49-bf01-ed17bbf2353e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:28:50.758707 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.758693    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bj9v\" (UniqueName: \"kubernetes.io/projected/1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef-kube-api-access-2bj9v\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:50.758779 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.758709    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61049872-551b-4d49-bf01-ed17bbf2353e-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:50.758779 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.758719    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:50.760185 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.760161    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61049872-551b-4d49-bf01-ed17bbf2353e-kube-api-access-lmwhr" pod "61049872-551b-4d49-bf01-ed17bbf2353e" (UID: "61049872-551b-4d49-bf01-ed17bbf2353e"). InnerVolumeSpecName "kube-api-access-lmwhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:50.860046 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.860005    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmwhr\" (UniqueName: \"kubernetes.io/projected/61049872-551b-4d49-bf01-ed17bbf2353e-kube-api-access-lmwhr\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:28:50.957006 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.956811    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:28:50.957624 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.957474    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:28:50.959192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:50.959168    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:28:51.211033 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.207331    5012 patch_prober.go:29] interesting pod/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 03 00:28:51.211033 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.207399    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms" podUID="ba93722b-9169-4628-9f61-baaef18a68a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 03 00:28:51.540984 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.537570    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh" event={"ID":"61049872-551b-4d49-bf01-ed17bbf2353e","Type":"ContainerDied","Data":"12e20ee337e9a589e29c9c3a815fc3c274a9c323a021f47e3c41bea8dde1a5d6"}
Nov 03 00:28:51.540984 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.537649    5012 scope.go:122] "RemoveContainer" containerID="fb83a95b13e1a19e932e7c63e0393d100a977aa4e95c2be01ff8280191141879"
Nov 03 00:28:51.540984 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.537753    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh"
Nov 03 00:28:51.556543 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.556516    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j"
Nov 03 00:28:51.567126 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.567079    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j" event={"ID":"1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef","Type":"ContainerDied","Data":"13ac4b73dc2b0948dfc1727807ccece19eec502578935d9ed2684cfa0f5d9f83"}
Nov 03 00:28:51.626759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.626419    5012 scope.go:122] "RemoveContainer" containerID="aca9b9149b2893bbd9f615f77ffcc4682d230b354926d0d0ded7e53de305d880"
Nov 03 00:28:51.636985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.636889    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh"]
Nov 03 00:28:51.642901 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.642877    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a43474-75b6674f47-wb9xh"]
Nov 03 00:28:51.755386 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.755336    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j"]
Nov 03 00:28:51.766377 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.762564    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-545986987f-z9x2j"]
Nov 03 00:28:51.956431 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.956396    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz"]
Nov 03 00:28:51.957412 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.957391    5012 cpu_manager.go:414] "containerMap: removing container" podUID="3dc8852e-27d6-4295-8580-cb92fcea2eaf" containerName="payload-processing"
Nov 03 00:28:51.957412 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.957412    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dc8852e-27d6-4295-8580-cb92fcea2eaf" containerName="payload-processing"
Nov 03 00:28:51.957564 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.957445    5012 cpu_manager.go:414] "containerMap: removing container" podUID="1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef" containerName="payload-pre-processing"
Nov 03 00:28:51.957564 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.957453    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef" containerName="payload-pre-processing"
Nov 03 00:28:51.957564 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.957490    5012 cpu_manager.go:414] "containerMap: removing container" podUID="860bdb4a-93f0-45de-b939-8471e7b7aecb" containerName="payload-pre-processing"
Nov 03 00:28:51.957564 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.957497    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="860bdb4a-93f0-45de-b939-8471e7b7aecb" containerName="payload-pre-processing"
Nov 03 00:28:51.957564 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.957511    5012 cpu_manager.go:414] "containerMap: removing container" podUID="61049872-551b-4d49-bf01-ed17bbf2353e" containerName="payload-processing"
Nov 03 00:28:51.957564 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.957518    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61049872-551b-4d49-bf01-ed17bbf2353e" containerName="payload-processing"
Nov 03 00:28:51.957751 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.957676    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="860bdb4a-93f0-45de-b939-8471e7b7aecb" containerName="payload-pre-processing"
Nov 03 00:28:51.957751 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.957691    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="61049872-551b-4d49-bf01-ed17bbf2353e" containerName="payload-processing"
Nov 03 00:28:51.957751 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.957708    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dc8852e-27d6-4295-8580-cb92fcea2eaf" containerName="payload-processing"
Nov 03 00:28:51.957751 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.957720    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef" containerName="payload-pre-processing"
Nov 03 00:28:51.963636 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.963617    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz"
Nov 03 00:28:51.977390 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.977334    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e37c9d\""
Nov 03 00:28:51.977762 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.977699    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e37c9d-dockercfg-5lb7s\""
Nov 03 00:28:51.980204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:51.980179    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz"]
Nov 03 00:28:52.119264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:52.119226    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz4x6\" (UniqueName: \"kubernetes.io/projected/48d61362-b777-4b66-84e2-03a542a6a761-kube-api-access-jz4x6\") pod \"payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz\" (UID: \"48d61362-b777-4b66-84e2-03a542a6a761\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz"
Nov 03 00:28:52.119414 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:52.119304    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48d61362-b777-4b66-84e2-03a542a6a761-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz\" (UID: \"48d61362-b777-4b66-84e2-03a542a6a761\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz"
Nov 03 00:28:52.119414 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:52.119312    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj"]
Nov 03 00:28:52.123717 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:52.123703    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj"
Nov 03 00:28:52.141599 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:52.141574    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj"]
Nov 03 00:28:52.211387 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:52.211301    5012 patch_prober.go:29] interesting pod/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 03 00:28:52.211387 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:52.211371    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms" podUID="ba93722b-9169-4628-9f61-baaef18a68a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 03 00:28:52.220190 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:52.220158    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7h9l\" (UniqueName: \"kubernetes.io/projected/b86d3c3e-36e1-406b-ae74-d8f846b60565-kube-api-access-x7h9l\") pod \"payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj\" (UID: \"b86d3c3e-36e1-406b-ae74-d8f846b60565\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj"
Nov 03 00:28:52.220304 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:52.220214    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz4x6\" (UniqueName: \"kubernetes.io/projected/48d61362-b777-4b66-84e2-03a542a6a761-kube-api-access-jz4x6\") pod \"payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz\" (UID: \"48d61362-b777-4b66-84e2-03a542a6a761\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz"
Nov 03 00:28:52.220304 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:52.220284    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48d61362-b777-4b66-84e2-03a542a6a761-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz\" (UID: \"48d61362-b777-4b66-84e2-03a542a6a761\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz"
Nov 03 00:28:52.220400 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:52.220385    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86d3c3e-36e1-406b-ae74-d8f846b60565-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj\" (UID: \"b86d3c3e-36e1-406b-ae74-d8f846b60565\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj"
Nov 03 00:28:52.221057 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:52.221037    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48d61362-b777-4b66-84e2-03a542a6a761-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz\" (UID: \"48d61362-b777-4b66-84e2-03a542a6a761\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz"
Nov 03 00:28:52.237919 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:52.237892    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz4x6\" (UniqueName: \"kubernetes.io/projected/48d61362-b777-4b66-84e2-03a542a6a761-kube-api-access-jz4x6\") pod \"payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz\" (UID: \"48d61362-b777-4b66-84e2-03a542a6a761\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz"
Nov 03 00:28:52.279477 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:52.279453    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz"
Nov 03 00:28:52.321247 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:52.321220    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86d3c3e-36e1-406b-ae74-d8f846b60565-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj\" (UID: \"b86d3c3e-36e1-406b-ae74-d8f846b60565\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj"
Nov 03 00:28:52.321404 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:52.321387    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7h9l\" (UniqueName: \"kubernetes.io/projected/b86d3c3e-36e1-406b-ae74-d8f846b60565-kube-api-access-x7h9l\") pod \"payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj\" (UID: \"b86d3c3e-36e1-406b-ae74-d8f846b60565\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj"
Nov 03 00:28:52.321954 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:52.321903    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86d3c3e-36e1-406b-ae74-d8f846b60565-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj\" (UID: \"b86d3c3e-36e1-406b-ae74-d8f846b60565\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj"
Nov 03 00:28:52.341277 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:52.338559    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7h9l\" (UniqueName: \"kubernetes.io/projected/b86d3c3e-36e1-406b-ae74-d8f846b60565-kube-api-access-x7h9l\") pod \"payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj\" (UID: \"b86d3c3e-36e1-406b-ae74-d8f846b60565\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj"
Nov 03 00:28:52.438078 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:52.438042    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj"
Nov 03 00:28:52.637253 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:52.637217    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz"]
Nov 03 00:28:52.638490 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:28:52.638455    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d61362_b777_4b66_84e2_03a542a6a761.slice/crio-df7926fb6e9c66da78682f9f21b64f234e6a6c36684d908c67d4bdc8bddd477f WatchSource:0}: Error finding container df7926fb6e9c66da78682f9f21b64f234e6a6c36684d908c67d4bdc8bddd477f: Status 404 returned error can't find the container with id df7926fb6e9c66da78682f9f21b64f234e6a6c36684d908c67d4bdc8bddd477f
Nov 03 00:28:52.761882 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:52.761853    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj"]
Nov 03 00:28:52.762530 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:28:52.762498    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86d3c3e_36e1_406b_ae74_d8f846b60565.slice/crio-c87e2ec241963bf794649721b00ea25cb3d62007495971ee5782158d03633841 WatchSource:0}: Error finding container c87e2ec241963bf794649721b00ea25cb3d62007495971ee5782158d03633841: Status 404 returned error can't find the container with id c87e2ec241963bf794649721b00ea25cb3d62007495971ee5782158d03633841
Nov 03 00:28:53.216961 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:53.207266    5012 patch_prober.go:29] interesting pod/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 03 00:28:53.216961 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:53.207320    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms" podUID="ba93722b-9169-4628-9f61-baaef18a68a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 03 00:28:53.245459 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:53.245424    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef" path="/var/lib/kubelet/pods/1e647aa0-04c1-4fd9-bb24-c0cc4a7d60ef/volumes"
Nov 03 00:28:53.245959 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:53.245937    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61049872-551b-4d49-bf01-ed17bbf2353e" path="/var/lib/kubelet/pods/61049872-551b-4d49-bf01-ed17bbf2353e/volumes"
Nov 03 00:28:53.563689 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:53.563596    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj" event={"ID":"b86d3c3e-36e1-406b-ae74-d8f846b60565","Type":"ContainerStarted","Data":"aecca86df24a981486d90a068b2c9061323f6ebd2d4e14314027838c0a42e7ef"}
Nov 03 00:28:53.563689 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:53.563655    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj" event={"ID":"b86d3c3e-36e1-406b-ae74-d8f846b60565","Type":"ContainerStarted","Data":"c87e2ec241963bf794649721b00ea25cb3d62007495971ee5782158d03633841"}
Nov 03 00:28:53.563689 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:53.563679    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj"
Nov 03 00:28:53.565397 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:53.565370    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz" event={"ID":"48d61362-b777-4b66-84e2-03a542a6a761","Type":"ContainerStarted","Data":"6c6042a62a30c313bd29fd60fcb879f43b3b8db79625256f4f02e031d31b0670"}
Nov 03 00:28:53.565525 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:53.565415    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz" event={"ID":"48d61362-b777-4b66-84e2-03a542a6a761","Type":"ContainerStarted","Data":"df7926fb6e9c66da78682f9f21b64f234e6a6c36684d908c67d4bdc8bddd477f"}
Nov 03 00:28:53.565525 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:53.565442    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz"
Nov 03 00:28:53.606555 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:53.606503    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj" podStartSLOduration=1.6064846830000001 podStartE2EDuration="1.606484683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:28:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:28:53.599122509 +0000 UTC m=+1416.478706557" watchObservedRunningTime="2026-11-03 00:28:53.606484683 +0000 UTC m=+1416.486068726"
Nov 03 00:28:53.650422 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:53.650369    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz" podStartSLOduration=2.6503577099999998 podStartE2EDuration="2.65035771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:28:53.645137806 +0000 UTC m=+1416.524721850" watchObservedRunningTime="2026-11-03 00:28:53.65035771 +0000 UTC m=+1416.529941754"
Nov 03 00:28:54.205281 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:54.205244    5012 patch_prober.go:29] interesting pod/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 03 00:28:54.205547 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:54.205517    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms" podUID="ba93722b-9169-4628-9f61-baaef18a68a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 03 00:28:54.375387 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:54.375357    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j"
Nov 03 00:28:55.208281 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:55.208252    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:28:55.208746 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:55.208732    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:28:55.209327 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:55.209306    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:28:56.283105 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.283061    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"]
Nov 03 00:28:56.289272 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.289244    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.306222 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.305396    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-4ae722-openshift-default-dockercfg-9l7b9\""
Nov 03 00:28:56.320345 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.320318    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"]
Nov 03 00:28:56.386204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.386173    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-workload-certs\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.386369 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.386227    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndstc\" (UniqueName: \"kubernetes.io/projected/59529c4e-6121-4fcf-94c2-4671ab16d5e2-kube-api-access-ndstc\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.386369 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.386251    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-podinfo\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.386499 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.386384    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-credential-socket\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.386499 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.386480    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istiod-ca-cert\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.386593 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.386540    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-token\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.386636 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.386622    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-data\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.386712 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.386692    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-workload-socket\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.386766 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.386729    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-envoy\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.488224 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.488186    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndstc\" (UniqueName: \"kubernetes.io/projected/59529c4e-6121-4fcf-94c2-4671ab16d5e2-kube-api-access-ndstc\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.488224 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.488223    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-podinfo\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.488500 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.488267    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-credential-socket\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.488500 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.488304    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istiod-ca-cert\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.488500 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.488368    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-token\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.488500 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.488406    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-data\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.488500 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.488443    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-workload-socket\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.488500 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.488473    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-envoy\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.488834 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.488532    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-workload-certs\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.488834 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.488802    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-data\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.489052 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.489025    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-credential-socket\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.489173 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.489148    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-workload-socket\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.489321 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.489299    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-workload-certs\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.489482 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.489457    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istiod-ca-cert\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.491127 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.491104    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-podinfo\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.491189 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.491136    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-envoy\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.505282 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.505259    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-token\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.505751 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.505731    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndstc\" (UniqueName: \"kubernetes.io/projected/59529c4e-6121-4fcf-94c2-4671ab16d5e2-kube-api-access-ndstc\") pod \"e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") " pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.620286 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.619951    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:56.997683 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:56.997657    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"]
Nov 03 00:28:56.997980 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:28:56.997957    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59529c4e_6121_4fcf_94c2_4671ab16d5e2.slice/crio-7b833565be823a8dd8973cefd8c9c5e700b06714147b1cc7dd4cecd913fdd117 WatchSource:0}: Error finding container 7b833565be823a8dd8973cefd8c9c5e700b06714147b1cc7dd4cecd913fdd117: Status 404 returned error can't find the container with id 7b833565be823a8dd8973cefd8c9c5e700b06714147b1cc7dd4cecd913fdd117
Nov 03 00:28:57.000685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:57.000653    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:28:57.000787 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:57.000716    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:28:57.000787 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:57.000742    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:28:57.703519 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:57.703476    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8" event={"ID":"59529c4e-6121-4fcf-94c2-4671ab16d5e2","Type":"ContainerStarted","Data":"1571366599c84892f1f34fef6db8205a271755c5d8e094464cf57108351e22d2"}
Nov 03 00:28:57.703885 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:57.703547    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8" event={"ID":"59529c4e-6121-4fcf-94c2-4671ab16d5e2","Type":"ContainerStarted","Data":"7b833565be823a8dd8973cefd8c9c5e700b06714147b1cc7dd4cecd913fdd117"}
Nov 03 00:28:57.755353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:57.755307    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8" podStartSLOduration=1.7552937640000001 podStartE2EDuration="1.755293764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:28:57.75075293 +0000 UTC m=+1420.630336972" watchObservedRunningTime="2026-11-03 00:28:57.755293764 +0000 UTC m=+1420.634877804"
Nov 03 00:28:58.621458 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:58.621417    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:58.627752 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:58.627719    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:58.711380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:58.711351    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:28:58.712316 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:28:58.712295    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:29:02.482664 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:02.482628    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp"]
Nov 03 00:29:02.491259 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:02.491227    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp"
Nov 03 00:29:02.507830 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:02.507805    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp"]
Nov 03 00:29:02.512365 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:02.512340    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-4ae722\""
Nov 03 00:29:02.572118 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:02.572088    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrpwh\" (UniqueName: \"kubernetes.io/projected/012c331f-034b-46f2-a430-ca3d7cdc9e4b-kube-api-access-zrpwh\") pod \"payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp\" (UID: \"012c331f-034b-46f2-a430-ca3d7cdc9e4b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp"
Nov 03 00:29:02.572298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:02.572274    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/012c331f-034b-46f2-a430-ca3d7cdc9e4b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp\" (UID: \"012c331f-034b-46f2-a430-ca3d7cdc9e4b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp"
Nov 03 00:29:02.674953 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:02.674898    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/012c331f-034b-46f2-a430-ca3d7cdc9e4b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp\" (UID: \"012c331f-034b-46f2-a430-ca3d7cdc9e4b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp"
Nov 03 00:29:02.675456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:02.675429    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrpwh\" (UniqueName: \"kubernetes.io/projected/012c331f-034b-46f2-a430-ca3d7cdc9e4b-kube-api-access-zrpwh\") pod \"payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp\" (UID: \"012c331f-034b-46f2-a430-ca3d7cdc9e4b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp"
Nov 03 00:29:02.676799 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:02.676774    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/012c331f-034b-46f2-a430-ca3d7cdc9e4b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp\" (UID: \"012c331f-034b-46f2-a430-ca3d7cdc9e4b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp"
Nov 03 00:29:02.698142 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:02.698112    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrpwh\" (UniqueName: \"kubernetes.io/projected/012c331f-034b-46f2-a430-ca3d7cdc9e4b-kube-api-access-zrpwh\") pod \"payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp\" (UID: \"012c331f-034b-46f2-a430-ca3d7cdc9e4b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp"
Nov 03 00:29:02.808474 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:02.808391    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp"
Nov 03 00:29:02.934378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:02.934347    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9"]
Nov 03 00:29:02.985886 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:02.985855    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj"]
Nov 03 00:29:03.131031 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:03.129588    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp"]
Nov 03 00:29:03.756311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:03.756278    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp" event={"ID":"012c331f-034b-46f2-a430-ca3d7cdc9e4b","Type":"ContainerStarted","Data":"606390c165991891bdc50130d6f6ec8d3b9d84299755637ba17d2622101a38af"}
Nov 03 00:29:03.756722 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:03.756330    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp" event={"ID":"012c331f-034b-46f2-a430-ca3d7cdc9e4b","Type":"ContainerStarted","Data":"2c26ffa62d03044f8f3c4663108a32cf065fd8ac506f1087cc1c4edf0267b0f2"}
Nov 03 00:29:03.756722 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:03.756376    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp"
Nov 03 00:29:03.793782 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:03.793734    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp" podStartSLOduration=1.793719786 podStartE2EDuration="1.793719786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:03.790488919 +0000 UTC m=+1426.670072962" watchObservedRunningTime="2026-11-03 00:29:03.793719786 +0000 UTC m=+1426.673303872"
Nov 03 00:29:04.653825 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:04.653794    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj"
Nov 03 00:29:04.654021 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:04.654008    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz"
Nov 03 00:29:06.965980 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:06.965269    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4"]
Nov 03 00:29:06.997807 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:06.996108    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4"]
Nov 03 00:29:06.997807 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:06.996266    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4"
Nov 03 00:29:07.010496 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.010366    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a43474-dockercfg-wvprk\""
Nov 03 00:29:07.010496 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.010463    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a43474\""
Nov 03 00:29:07.060685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.060446    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plr7q\" (UniqueName: \"kubernetes.io/projected/e2bd83dc-8985-4d62-86eb-4fb74ec20dd7-kube-api-access-plr7q\") pod \"payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4\" (UID: \"e2bd83dc-8985-4d62-86eb-4fb74ec20dd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4"
Nov 03 00:29:07.060685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.060515    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2bd83dc-8985-4d62-86eb-4fb74ec20dd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4\" (UID: \"e2bd83dc-8985-4d62-86eb-4fb74ec20dd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4"
Nov 03 00:29:07.124531 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.124502    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk"]
Nov 03 00:29:07.128872 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.128850    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk"
Nov 03 00:29:07.145040 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.145016    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk"]
Nov 03 00:29:07.171890 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.171851    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plr7q\" (UniqueName: \"kubernetes.io/projected/e2bd83dc-8985-4d62-86eb-4fb74ec20dd7-kube-api-access-plr7q\") pod \"payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4\" (UID: \"e2bd83dc-8985-4d62-86eb-4fb74ec20dd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4"
Nov 03 00:29:07.172049 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.171939    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2bd83dc-8985-4d62-86eb-4fb74ec20dd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4\" (UID: \"e2bd83dc-8985-4d62-86eb-4fb74ec20dd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4"
Nov 03 00:29:07.173181 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.173155    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2bd83dc-8985-4d62-86eb-4fb74ec20dd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4\" (UID: \"e2bd83dc-8985-4d62-86eb-4fb74ec20dd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4"
Nov 03 00:29:07.189409 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.189388    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plr7q\" (UniqueName: \"kubernetes.io/projected/e2bd83dc-8985-4d62-86eb-4fb74ec20dd7-kube-api-access-plr7q\") pod \"payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4\" (UID: \"e2bd83dc-8985-4d62-86eb-4fb74ec20dd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4"
Nov 03 00:29:07.273382 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.273327    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-przph\" (UniqueName: \"kubernetes.io/projected/6029bbe2-1d00-4960-8555-f5e6e1588bfa-kube-api-access-przph\") pod \"payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk\" (UID: \"6029bbe2-1d00-4960-8555-f5e6e1588bfa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk"
Nov 03 00:29:07.273547 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.273517    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6029bbe2-1d00-4960-8555-f5e6e1588bfa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk\" (UID: \"6029bbe2-1d00-4960-8555-f5e6e1588bfa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk"
Nov 03 00:29:07.364157 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:29:07.364125    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:29:07.371466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.371445    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4"
Nov 03 00:29:07.374549 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.374503    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6029bbe2-1d00-4960-8555-f5e6e1588bfa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk\" (UID: \"6029bbe2-1d00-4960-8555-f5e6e1588bfa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk"
Nov 03 00:29:07.374658 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.374639    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-przph\" (UniqueName: \"kubernetes.io/projected/6029bbe2-1d00-4960-8555-f5e6e1588bfa-kube-api-access-przph\") pod \"payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk\" (UID: \"6029bbe2-1d00-4960-8555-f5e6e1588bfa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk"
Nov 03 00:29:07.375366 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.375347    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6029bbe2-1d00-4960-8555-f5e6e1588bfa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk\" (UID: \"6029bbe2-1d00-4960-8555-f5e6e1588bfa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk"
Nov 03 00:29:07.391748 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.391725    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-przph\" (UniqueName: \"kubernetes.io/projected/6029bbe2-1d00-4960-8555-f5e6e1588bfa-kube-api-access-przph\") pod \"payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk\" (UID: \"6029bbe2-1d00-4960-8555-f5e6e1588bfa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk"
Nov 03 00:29:07.452211 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.444451    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk"
Nov 03 00:29:07.669970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.669940    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4"]
Nov 03 00:29:07.670345 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:29:07.670319    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2bd83dc_8985_4d62_86eb_4fb74ec20dd7.slice/crio-d3229a744ce01a4500f5935c1533682a6b5addb546332c44a44d84852bac1ff9 WatchSource:0}: Error finding container d3229a744ce01a4500f5935c1533682a6b5addb546332c44a44d84852bac1ff9: Status 404 returned error can't find the container with id d3229a744ce01a4500f5935c1533682a6b5addb546332c44a44d84852bac1ff9
Nov 03 00:29:07.758461 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:29:07.758425    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6029bbe2_1d00_4960_8555_f5e6e1588bfa.slice/crio-a3363fda0126e640363b9f383918fea1c31f84d87b80da7431bbc267f8f02108 WatchSource:0}: Error finding container a3363fda0126e640363b9f383918fea1c31f84d87b80da7431bbc267f8f02108: Status 404 returned error can't find the container with id a3363fda0126e640363b9f383918fea1c31f84d87b80da7431bbc267f8f02108
Nov 03 00:29:07.759050 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.759032    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk"]
Nov 03 00:29:07.797390 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.797355    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk" event={"ID":"6029bbe2-1d00-4960-8555-f5e6e1588bfa","Type":"ContainerStarted","Data":"a3363fda0126e640363b9f383918fea1c31f84d87b80da7431bbc267f8f02108"}
Nov 03 00:29:07.799680 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.799642    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4" event={"ID":"e2bd83dc-8985-4d62-86eb-4fb74ec20dd7","Type":"ContainerStarted","Data":"f034887dfdfdbf79ce737df951d092fc4280439972cf0961a449a081bbe48997"}
Nov 03 00:29:07.799813 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.799698    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4" event={"ID":"e2bd83dc-8985-4d62-86eb-4fb74ec20dd7","Type":"ContainerStarted","Data":"d3229a744ce01a4500f5935c1533682a6b5addb546332c44a44d84852bac1ff9"}
Nov 03 00:29:07.799813 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.799720    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4"
Nov 03 00:29:07.845595 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.845540    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4" podStartSLOduration=1.845525214 podStartE2EDuration="1.845525214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:07.843825689 +0000 UTC m=+1430.723409731" watchObservedRunningTime="2026-11-03 00:29:07.845525214 +0000 UTC m=+1430.725109256"
Nov 03 00:29:07.935584 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.935501    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9" podUID="09fa3da2-974b-4530-aed4-d64a3223ccbc" containerName="payload-processing" containerID="cri-o://695c2d16d9242fa99e8cbfb2b565396a80681b36b958c54cf3c8d5d901a9ac1b" gracePeriod=25
Nov 03 00:29:07.986720 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:07.986653    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj" podUID="3c081efc-84f9-4246-abbf-947475128655" containerName="payload-pre-processing" containerID="cri-o://c2216f4da2e346de44a6bc85776f8d3d4969287fa827bda9e6d114314939f552" gracePeriod=25
Nov 03 00:29:08.287185 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.287159    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9"
Nov 03 00:29:08.342123 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.342098    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj"
Nov 03 00:29:08.387770 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.387742    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09fa3da2-974b-4530-aed4-d64a3223ccbc-kube-api-access-72pd5\" (UniqueName: \"kubernetes.io/projected/09fa3da2-974b-4530-aed4-d64a3223ccbc-kube-api-access-72pd5\") pod \"09fa3da2-974b-4530-aed4-d64a3223ccbc\" (UID: \"09fa3da2-974b-4530-aed4-d64a3223ccbc\") "
Nov 03 00:29:08.388013 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.387995    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09fa3da2-974b-4530-aed4-d64a3223ccbc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09fa3da2-974b-4530-aed4-d64a3223ccbc-plugins-config-volume\") pod \"09fa3da2-974b-4530-aed4-d64a3223ccbc\" (UID: \"09fa3da2-974b-4530-aed4-d64a3223ccbc\") "
Nov 03 00:29:08.388153 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.388133    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c081efc-84f9-4246-abbf-947475128655-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c081efc-84f9-4246-abbf-947475128655-plugins-config-volume\") pod \"3c081efc-84f9-4246-abbf-947475128655\" (UID: \"3c081efc-84f9-4246-abbf-947475128655\") "
Nov 03 00:29:08.388422 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.388323    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c081efc-84f9-4246-abbf-947475128655-kube-api-access-7hc8l\" (UniqueName: \"kubernetes.io/projected/3c081efc-84f9-4246-abbf-947475128655-kube-api-access-7hc8l\") pod \"3c081efc-84f9-4246-abbf-947475128655\" (UID: \"3c081efc-84f9-4246-abbf-947475128655\") "
Nov 03 00:29:08.389526 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.389466    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c081efc-84f9-4246-abbf-947475128655-plugins-config-volume" pod "3c081efc-84f9-4246-abbf-947475128655" (UID: "3c081efc-84f9-4246-abbf-947475128655"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:29:08.390155 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.390125    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09fa3da2-974b-4530-aed4-d64a3223ccbc-plugins-config-volume" pod "09fa3da2-974b-4530-aed4-d64a3223ccbc" (UID: "09fa3da2-974b-4530-aed4-d64a3223ccbc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:29:08.391407 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.391381    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09fa3da2-974b-4530-aed4-d64a3223ccbc-kube-api-access-72pd5" pod "09fa3da2-974b-4530-aed4-d64a3223ccbc" (UID: "09fa3da2-974b-4530-aed4-d64a3223ccbc"). InnerVolumeSpecName "kube-api-access-72pd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:29:08.391503 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.391431    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c081efc-84f9-4246-abbf-947475128655-kube-api-access-7hc8l" pod "3c081efc-84f9-4246-abbf-947475128655" (UID: "3c081efc-84f9-4246-abbf-947475128655"). InnerVolumeSpecName "kube-api-access-7hc8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:29:08.497801 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.497717    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72pd5\" (UniqueName: \"kubernetes.io/projected/09fa3da2-974b-4530-aed4-d64a3223ccbc-kube-api-access-72pd5\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:29:08.497801 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.497758    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09fa3da2-974b-4530-aed4-d64a3223ccbc-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:29:08.497801 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.497772    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c081efc-84f9-4246-abbf-947475128655-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:29:08.497801 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.497784    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hc8l\" (UniqueName: \"kubernetes.io/projected/3c081efc-84f9-4246-abbf-947475128655-kube-api-access-7hc8l\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:29:08.810596 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.810504    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk" event={"ID":"6029bbe2-1d00-4960-8555-f5e6e1588bfa","Type":"ContainerStarted","Data":"4b8770a7b9818c54e66a9903fa839da48607d492fe08995b79544048cc564f3a"}
Nov 03 00:29:08.810756 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.810607    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk"
Nov 03 00:29:08.812299 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.812266    5012 generic.go:356] "Generic (PLEG): container finished" podID="09fa3da2-974b-4530-aed4-d64a3223ccbc" containerID="695c2d16d9242fa99e8cbfb2b565396a80681b36b958c54cf3c8d5d901a9ac1b" exitCode=0
Nov 03 00:29:08.812421 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.812325    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9"
Nov 03 00:29:08.812421 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.812351    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9" event={"ID":"09fa3da2-974b-4530-aed4-d64a3223ccbc","Type":"ContainerDied","Data":"695c2d16d9242fa99e8cbfb2b565396a80681b36b958c54cf3c8d5d901a9ac1b"}
Nov 03 00:29:08.812421 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.812406    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9" event={"ID":"09fa3da2-974b-4530-aed4-d64a3223ccbc","Type":"ContainerDied","Data":"475adbda1ab6dc21af67d25233fbc754a2e6debc3cda3b64d35b9bda7c9a1bb6"}
Nov 03 00:29:08.812584 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.812436    5012 scope.go:122] "RemoveContainer" containerID="695c2d16d9242fa99e8cbfb2b565396a80681b36b958c54cf3c8d5d901a9ac1b"
Nov 03 00:29:08.814107 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.814077    5012 generic.go:356] "Generic (PLEG): container finished" podID="3c081efc-84f9-4246-abbf-947475128655" containerID="c2216f4da2e346de44a6bc85776f8d3d4969287fa827bda9e6d114314939f552" exitCode=0
Nov 03 00:29:08.814194 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.814171    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj"
Nov 03 00:29:08.814262 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.814218    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj" event={"ID":"3c081efc-84f9-4246-abbf-947475128655","Type":"ContainerDied","Data":"c2216f4da2e346de44a6bc85776f8d3d4969287fa827bda9e6d114314939f552"}
Nov 03 00:29:08.814262 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.814251    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj" event={"ID":"3c081efc-84f9-4246-abbf-947475128655","Type":"ContainerDied","Data":"08c28b91040653cc4bdf2116e410b56595fb13e7a556979c7ab9a605568d879a"}
Nov 03 00:29:08.827090 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.827063    5012 scope.go:122] "RemoveContainer" containerID="695c2d16d9242fa99e8cbfb2b565396a80681b36b958c54cf3c8d5d901a9ac1b"
Nov 03 00:29:08.827442 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:29:08.827416    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"695c2d16d9242fa99e8cbfb2b565396a80681b36b958c54cf3c8d5d901a9ac1b\": container with ID starting with 695c2d16d9242fa99e8cbfb2b565396a80681b36b958c54cf3c8d5d901a9ac1b not found: ID does not exist" containerID="695c2d16d9242fa99e8cbfb2b565396a80681b36b958c54cf3c8d5d901a9ac1b"
Nov 03 00:29:08.827556 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.827535    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"695c2d16d9242fa99e8cbfb2b565396a80681b36b958c54cf3c8d5d901a9ac1b"} err="failed to get container status \"695c2d16d9242fa99e8cbfb2b565396a80681b36b958c54cf3c8d5d901a9ac1b\": rpc error: code = NotFound desc = could not find container \"695c2d16d9242fa99e8cbfb2b565396a80681b36b958c54cf3c8d5d901a9ac1b\": container with ID starting with 695c2d16d9242fa99e8cbfb2b565396a80681b36b958c54cf3c8d5d901a9ac1b not found: ID does not exist"
Nov 03 00:29:08.827653 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.827636    5012 scope.go:122] "RemoveContainer" containerID="c2216f4da2e346de44a6bc85776f8d3d4969287fa827bda9e6d114314939f552"
Nov 03 00:29:08.837594 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.837574    5012 scope.go:122] "RemoveContainer" containerID="c2216f4da2e346de44a6bc85776f8d3d4969287fa827bda9e6d114314939f552"
Nov 03 00:29:08.837920 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:29:08.837867    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2216f4da2e346de44a6bc85776f8d3d4969287fa827bda9e6d114314939f552\": container with ID starting with c2216f4da2e346de44a6bc85776f8d3d4969287fa827bda9e6d114314939f552 not found: ID does not exist" containerID="c2216f4da2e346de44a6bc85776f8d3d4969287fa827bda9e6d114314939f552"
Nov 03 00:29:08.838038 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.837943    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2216f4da2e346de44a6bc85776f8d3d4969287fa827bda9e6d114314939f552"} err="failed to get container status \"c2216f4da2e346de44a6bc85776f8d3d4969287fa827bda9e6d114314939f552\": rpc error: code = NotFound desc = could not find container \"c2216f4da2e346de44a6bc85776f8d3d4969287fa827bda9e6d114314939f552\": container with ID starting with c2216f4da2e346de44a6bc85776f8d3d4969287fa827bda9e6d114314939f552 not found: ID does not exist"
Nov 03 00:29:08.854923 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.848748    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk" podStartSLOduration=1.848734238 podStartE2EDuration="1.848734238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:08.847841989 +0000 UTC m=+1431.727426032" watchObservedRunningTime="2026-11-03 00:29:08.848734238 +0000 UTC m=+1431.728318287"
Nov 03 00:29:08.884697 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.884676    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9"]
Nov 03 00:29:08.890741 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.890717    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-59d8d66bc9-c6nk9"]
Nov 03 00:29:08.920362 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.920165    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj"]
Nov 03 00:29:08.924566 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:08.924547    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-5fbdd44c5-c9blj"]
Nov 03 00:29:09.246306 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:09.246272    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09fa3da2-974b-4530-aed4-d64a3223ccbc" path="/var/lib/kubelet/pods/09fa3da2-974b-4530-aed4-d64a3223ccbc/volumes"
Nov 03 00:29:09.246791 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:09.246774    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c081efc-84f9-4246-abbf-947475128655" path="/var/lib/kubelet/pods/3c081efc-84f9-4246-abbf-947475128655/volumes"
Nov 03 00:29:14.769361 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:14.769329    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp"
Nov 03 00:29:18.757999 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:18.757964    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff"]
Nov 03 00:29:18.758432 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:18.758168    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff" podUID="7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234" containerName="payload-processing" containerID="cri-o://90b218e6ae0dfc758f6167290acb92134c0d09bf127927cd39aba0b54c448baf" gracePeriod=30
Nov 03 00:29:18.832955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:18.826452    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67"]
Nov 03 00:29:18.832955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:18.826689    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67" podUID="dbd7242d-9d05-487a-8d54-78667f412229" containerName="payload-pre-processing" containerID="cri-o://75daaaa76d5c7ec2a7d05eb70c4143df5c1d61f41a08ff51c2cbd3f37a7f2bb7" gracePeriod=30
Nov 03 00:29:18.832955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:18.829501    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4"
Nov 03 00:29:18.942685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:18.942656    5012 generic.go:356] "Generic (PLEG): container finished" podID="7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234" containerID="90b218e6ae0dfc758f6167290acb92134c0d09bf127927cd39aba0b54c448baf" exitCode=0
Nov 03 00:29:18.942961 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:18.942943    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff" event={"ID":"7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234","Type":"ContainerDied","Data":"90b218e6ae0dfc758f6167290acb92134c0d09bf127927cd39aba0b54c448baf"}
Nov 03 00:29:19.374124 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:19.374096    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff"
Nov 03 00:29:19.461063 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:19.461039    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67"
Nov 03 00:29:19.468535 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:19.468503    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234-kube-api-access-tqkk5\" (UniqueName: \"kubernetes.io/projected/7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234-kube-api-access-tqkk5\") pod \"7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234\" (UID: \"7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234\") "
Nov 03 00:29:19.468822 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:19.468800    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234-plugins-config-volume\") pod \"7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234\" (UID: \"7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234\") "
Nov 03 00:29:19.469247 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:19.469203    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234-plugins-config-volume" pod "7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234" (UID: "7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:29:19.469634 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:19.469612    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:29:19.470751 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:19.470728    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234-kube-api-access-tqkk5" pod "7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234" (UID: "7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234"). InnerVolumeSpecName "kube-api-access-tqkk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:29:19.570853 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:19.570777    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd7242d-9d05-487a-8d54-78667f412229-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd7242d-9d05-487a-8d54-78667f412229-plugins-config-volume\") pod \"dbd7242d-9d05-487a-8d54-78667f412229\" (UID: \"dbd7242d-9d05-487a-8d54-78667f412229\") "
Nov 03 00:29:19.570853 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:19.570842    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd7242d-9d05-487a-8d54-78667f412229-kube-api-access-vkczf\" (UniqueName: \"kubernetes.io/projected/dbd7242d-9d05-487a-8d54-78667f412229-kube-api-access-vkczf\") pod \"dbd7242d-9d05-487a-8d54-78667f412229\" (UID: \"dbd7242d-9d05-487a-8d54-78667f412229\") "
Nov 03 00:29:19.571203 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:19.571179    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd7242d-9d05-487a-8d54-78667f412229-plugins-config-volume" pod "dbd7242d-9d05-487a-8d54-78667f412229" (UID: "dbd7242d-9d05-487a-8d54-78667f412229"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:29:19.571284 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:19.571270    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqkk5\" (UniqueName: \"kubernetes.io/projected/7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234-kube-api-access-tqkk5\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:29:19.572936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:19.572897    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd7242d-9d05-487a-8d54-78667f412229-kube-api-access-vkczf" pod "dbd7242d-9d05-487a-8d54-78667f412229" (UID: "dbd7242d-9d05-487a-8d54-78667f412229"). InnerVolumeSpecName "kube-api-access-vkczf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:29:19.674367 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:19.673078    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd7242d-9d05-487a-8d54-78667f412229-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:29:19.674367 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:19.673114    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkczf\" (UniqueName: \"kubernetes.io/projected/dbd7242d-9d05-487a-8d54-78667f412229-kube-api-access-vkczf\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:29:19.827118 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:19.827040    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk"
Nov 03 00:29:19.952066 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:19.952025    5012 generic.go:356] "Generic (PLEG): container finished" podID="dbd7242d-9d05-487a-8d54-78667f412229" containerID="75daaaa76d5c7ec2a7d05eb70c4143df5c1d61f41a08ff51c2cbd3f37a7f2bb7" exitCode=0
Nov 03 00:29:19.952233 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:19.952071    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67"
Nov 03 00:29:19.952233 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:19.952106    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67" event={"ID":"dbd7242d-9d05-487a-8d54-78667f412229","Type":"ContainerDied","Data":"75daaaa76d5c7ec2a7d05eb70c4143df5c1d61f41a08ff51c2cbd3f37a7f2bb7"}
Nov 03 00:29:19.952233 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:19.952159    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67" event={"ID":"dbd7242d-9d05-487a-8d54-78667f412229","Type":"ContainerDied","Data":"89aacef159632237fb1cd1dc6205359025fdcc2a288e0dd9444d65d31c32643f"}
Nov 03 00:29:19.952233 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:19.952188    5012 scope.go:122] "RemoveContainer" containerID="75daaaa76d5c7ec2a7d05eb70c4143df5c1d61f41a08ff51c2cbd3f37a7f2bb7"
Nov 03 00:29:19.954254 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:19.954232    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff" event={"ID":"7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234","Type":"ContainerDied","Data":"4876f429898ccebabdbed39e44b4f384df87c5c7aad3464816e77bbddb761e03"}
Nov 03 00:29:19.954355 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:19.954256    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff"
Nov 03 00:29:19.965658 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:19.965618    5012 scope.go:122] "RemoveContainer" containerID="75daaaa76d5c7ec2a7d05eb70c4143df5c1d61f41a08ff51c2cbd3f37a7f2bb7"
Nov 03 00:29:19.965999 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:29:19.965967    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75daaaa76d5c7ec2a7d05eb70c4143df5c1d61f41a08ff51c2cbd3f37a7f2bb7\": container with ID starting with 75daaaa76d5c7ec2a7d05eb70c4143df5c1d61f41a08ff51c2cbd3f37a7f2bb7 not found: ID does not exist" containerID="75daaaa76d5c7ec2a7d05eb70c4143df5c1d61f41a08ff51c2cbd3f37a7f2bb7"
Nov 03 00:29:19.966097 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:19.966005    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75daaaa76d5c7ec2a7d05eb70c4143df5c1d61f41a08ff51c2cbd3f37a7f2bb7"} err="failed to get container status \"75daaaa76d5c7ec2a7d05eb70c4143df5c1d61f41a08ff51c2cbd3f37a7f2bb7\": rpc error: code = NotFound desc = could not find container \"75daaaa76d5c7ec2a7d05eb70c4143df5c1d61f41a08ff51c2cbd3f37a7f2bb7\": container with ID starting with 75daaaa76d5c7ec2a7d05eb70c4143df5c1d61f41a08ff51c2cbd3f37a7f2bb7 not found: ID does not exist"
Nov 03 00:29:19.966097 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:19.966036    5012 scope.go:122] "RemoveContainer" containerID="90b218e6ae0dfc758f6167290acb92134c0d09bf127927cd39aba0b54c448baf"
Nov 03 00:29:20.016760 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:20.016728    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67"]
Nov 03 00:29:20.021392 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:20.021364    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3437aa-fd6f56679-9fl67"]
Nov 03 00:29:20.056468 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:20.056438    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff"]
Nov 03 00:29:20.060810 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:20.060784    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3437aa-5864bf56cb-2cfff"]
Nov 03 00:29:21.284958 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.284899    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234" path="/var/lib/kubelet/pods/7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234/volumes"
Nov 03 00:29:21.285748 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.285724    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd7242d-9d05-487a-8d54-78667f412229" path="/var/lib/kubelet/pods/dbd7242d-9d05-487a-8d54-78667f412229/volumes"
Nov 03 00:29:21.535688 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.535582    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7"]
Nov 03 00:29:21.536623 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.536595    5012 cpu_manager.go:414] "containerMap: removing container" podUID="7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234" containerName="payload-processing"
Nov 03 00:29:21.536623 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.536620    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234" containerName="payload-processing"
Nov 03 00:29:21.536814 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.536639    5012 cpu_manager.go:414] "containerMap: removing container" podUID="3c081efc-84f9-4246-abbf-947475128655" containerName="payload-pre-processing"
Nov 03 00:29:21.536814 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.536648    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c081efc-84f9-4246-abbf-947475128655" containerName="payload-pre-processing"
Nov 03 00:29:21.536814 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.536701    5012 cpu_manager.go:414] "containerMap: removing container" podUID="09fa3da2-974b-4530-aed4-d64a3223ccbc" containerName="payload-processing"
Nov 03 00:29:21.536814 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.536709    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09fa3da2-974b-4530-aed4-d64a3223ccbc" containerName="payload-processing"
Nov 03 00:29:21.536814 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.536738    5012 cpu_manager.go:414] "containerMap: removing container" podUID="dbd7242d-9d05-487a-8d54-78667f412229" containerName="payload-pre-processing"
Nov 03 00:29:21.536814 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.536746    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd7242d-9d05-487a-8d54-78667f412229" containerName="payload-pre-processing"
Nov 03 00:29:21.537180 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.536932    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ef4f6ca-5dc0-4168-bf5e-2a9a3efd3234" containerName="payload-processing"
Nov 03 00:29:21.537180 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.536966    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c081efc-84f9-4246-abbf-947475128655" containerName="payload-pre-processing"
Nov 03 00:29:21.537180 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.536987    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="09fa3da2-974b-4530-aed4-d64a3223ccbc" containerName="payload-processing"
Nov 03 00:29:21.537180 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.537001    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd7242d-9d05-487a-8d54-78667f412229" containerName="payload-pre-processing"
Nov 03 00:29:21.548352 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.548323    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7"]
Nov 03 00:29:21.548482 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.548456    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7"
Nov 03 00:29:21.585038 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.585008    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-54f6d8-dockercfg-448md\""
Nov 03 00:29:21.585770 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.585705    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-54f6d8\""
Nov 03 00:29:21.666702 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.666587    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4"]
Nov 03 00:29:21.672415 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.672107    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4"
Nov 03 00:29:21.726433 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.726334    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4"]
Nov 03 00:29:21.737117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.736320    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faff8e48-0925-4faf-84be-a5529573e717-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7\" (UID: \"faff8e48-0925-4faf-84be-a5529573e717\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7"
Nov 03 00:29:21.737117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.736382    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t28px\" (UniqueName: \"kubernetes.io/projected/faff8e48-0925-4faf-84be-a5529573e717-kube-api-access-t28px\") pod \"payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7\" (UID: \"faff8e48-0925-4faf-84be-a5529573e717\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7"
Nov 03 00:29:21.838430 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.838336    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faff8e48-0925-4faf-84be-a5529573e717-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7\" (UID: \"faff8e48-0925-4faf-84be-a5529573e717\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7"
Nov 03 00:29:21.838430 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.838395    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t28px\" (UniqueName: \"kubernetes.io/projected/faff8e48-0925-4faf-84be-a5529573e717-kube-api-access-t28px\") pod \"payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7\" (UID: \"faff8e48-0925-4faf-84be-a5529573e717\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7"
Nov 03 00:29:21.838671 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.838487    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/933c9a50-87ce-4b10-ba38-3fcdd105b7f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4\" (UID: \"933c9a50-87ce-4b10-ba38-3fcdd105b7f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4"
Nov 03 00:29:21.838671 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.838546    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75lzn\" (UniqueName: \"kubernetes.io/projected/933c9a50-87ce-4b10-ba38-3fcdd105b7f7-kube-api-access-75lzn\") pod \"payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4\" (UID: \"933c9a50-87ce-4b10-ba38-3fcdd105b7f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4"
Nov 03 00:29:21.839094 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.839075    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faff8e48-0925-4faf-84be-a5529573e717-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7\" (UID: \"faff8e48-0925-4faf-84be-a5529573e717\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7"
Nov 03 00:29:21.855991 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.855953    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t28px\" (UniqueName: \"kubernetes.io/projected/faff8e48-0925-4faf-84be-a5529573e717-kube-api-access-t28px\") pod \"payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7\" (UID: \"faff8e48-0925-4faf-84be-a5529573e717\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7"
Nov 03 00:29:21.908419 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.908382    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7"
Nov 03 00:29:21.940268 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.940240    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/933c9a50-87ce-4b10-ba38-3fcdd105b7f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4\" (UID: \"933c9a50-87ce-4b10-ba38-3fcdd105b7f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4"
Nov 03 00:29:21.940376 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.940308    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75lzn\" (UniqueName: \"kubernetes.io/projected/933c9a50-87ce-4b10-ba38-3fcdd105b7f7-kube-api-access-75lzn\") pod \"payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4\" (UID: \"933c9a50-87ce-4b10-ba38-3fcdd105b7f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4"
Nov 03 00:29:21.941144 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.941117    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/933c9a50-87ce-4b10-ba38-3fcdd105b7f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4\" (UID: \"933c9a50-87ce-4b10-ba38-3fcdd105b7f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4"
Nov 03 00:29:21.957503 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:21.957479    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75lzn\" (UniqueName: \"kubernetes.io/projected/933c9a50-87ce-4b10-ba38-3fcdd105b7f7-kube-api-access-75lzn\") pod \"payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4\" (UID: \"933c9a50-87ce-4b10-ba38-3fcdd105b7f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4"
Nov 03 00:29:22.013818 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:22.013168    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4"
Nov 03 00:29:22.220300 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:22.220271    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7"]
Nov 03 00:29:22.341286 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:29:22.341260    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933c9a50_87ce_4b10_ba38_3fcdd105b7f7.slice/crio-4a319c8d7c9a648700299d678fdea0322a52aefb870cd098a178776244541f6e WatchSource:0}: Error finding container 4a319c8d7c9a648700299d678fdea0322a52aefb870cd098a178776244541f6e: Status 404 returned error can't find the container with id 4a319c8d7c9a648700299d678fdea0322a52aefb870cd098a178776244541f6e
Nov 03 00:29:22.341585 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:22.341384    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4"]
Nov 03 00:29:23.007934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:22.998716    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7" event={"ID":"faff8e48-0925-4faf-84be-a5529573e717","Type":"ContainerStarted","Data":"f1f605098d5fd45d22403730b0f7e3148ca93b0da755cd303e9bd2f559343a54"}
Nov 03 00:29:23.007934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:22.998850    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7" event={"ID":"faff8e48-0925-4faf-84be-a5529573e717","Type":"ContainerStarted","Data":"203b95e3aff314ff088369e5dc0516e7db0b9bb99b32530c74a0b6432fdeed24"}
Nov 03 00:29:23.007934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:22.999061    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7"
Nov 03 00:29:23.007934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:23.002130    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4" event={"ID":"933c9a50-87ce-4b10-ba38-3fcdd105b7f7","Type":"ContainerStarted","Data":"485e256f2acb887dfd2380386c79b28b685c743510d6cae68d58ebe3704b2d2a"}
Nov 03 00:29:23.007934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:23.002174    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4" event={"ID":"933c9a50-87ce-4b10-ba38-3fcdd105b7f7","Type":"ContainerStarted","Data":"4a319c8d7c9a648700299d678fdea0322a52aefb870cd098a178776244541f6e"}
Nov 03 00:29:23.007934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:23.002793    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4"
Nov 03 00:29:23.052823 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:23.052749    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7" podStartSLOduration=2.052731775 podStartE2EDuration="2.052731775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:23.048561585 +0000 UTC m=+1445.928145632" watchObservedRunningTime="2026-11-03 00:29:23.052731775 +0000 UTC m=+1445.932315819"
Nov 03 00:29:23.106278 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:23.106221    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4" podStartSLOduration=2.106207914 podStartE2EDuration="2.106207914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:23.098947618 +0000 UTC m=+1445.978531656" watchObservedRunningTime="2026-11-03 00:29:23.106207914 +0000 UTC m=+1445.985791957"
Nov 03 00:29:25.325095 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:25.325058    5012 patch_prober.go:29] interesting pod/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:29:25.325503 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:25.325117    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz" podUID="5e8e36a8-3445-41ab-a889-c529c3e707c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:29:28.080799 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:28.080767    5012 generic.go:356] "Generic (PLEG): container finished" podID="5e8e36a8-3445-41ab-a889-c529c3e707c9" containerID="cd96a43eed3105271ace6242ac2fc21431a985189829b6a578bf4b6d381089ab" exitCode=0
Nov 03 00:29:28.081249 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:28.080838    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz" event={"ID":"5e8e36a8-3445-41ab-a889-c529c3e707c9","Type":"ContainerDied","Data":"cd96a43eed3105271ace6242ac2fc21431a985189829b6a578bf4b6d381089ab"}
Nov 03 00:29:28.081249 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:28.081200    5012 scope.go:122] "RemoveContainer" containerID="cd96a43eed3105271ace6242ac2fc21431a985189829b6a578bf4b6d381089ab"
Nov 03 00:29:28.082642 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:28.082618    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:29:28.082710 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:28.082681    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:29:28.082748 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:28.082712    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:29:29.249574 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:29.238449    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz" event={"ID":"5e8e36a8-3445-41ab-a889-c529c3e707c9","Type":"ContainerStarted","Data":"24376c343bb977be3c828b2c83525432a46508f6e61eed1c86441a12f6b1e155"}
Nov 03 00:29:29.465099 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:29.465063    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:29:29.465271 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:29.465243    5012 patch_prober.go:29] interesting pod/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Nov 03 00:29:29.465327 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:29.465302    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz" podUID="5e8e36a8-3445-41ab-a889-c529c3e707c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Nov 03 00:29:30.464618 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:30.464581    5012 patch_prober.go:29] interesting pod/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Nov 03 00:29:30.465070 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:30.464648    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz" podUID="5e8e36a8-3445-41ab-a889-c529c3e707c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Nov 03 00:29:31.465202 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:31.465166    5012 patch_prober.go:29] interesting pod/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Nov 03 00:29:31.465595 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:31.465217    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz" podUID="5e8e36a8-3445-41ab-a889-c529c3e707c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Nov 03 00:29:31.575954 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:31.575920    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j"]
Nov 03 00:29:32.464818 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:32.464776    5012 patch_prober.go:29] interesting pod/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Nov 03 00:29:32.465026 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:32.464829    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz" podUID="5e8e36a8-3445-41ab-a889-c529c3e707c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Nov 03 00:29:33.464316 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:33.464274    5012 patch_prober.go:29] interesting pod/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Nov 03 00:29:33.464727 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:33.464337    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz" podUID="5e8e36a8-3445-41ab-a889-c529c3e707c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Nov 03 00:29:34.014761 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:34.014734    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7"
Nov 03 00:29:34.693880 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:34.693827    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:29:34.694507 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:34.694471    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:29:34.695607 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:34.695585    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:29:35.023312 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:35.023245    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4"
Nov 03 00:29:36.251587 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:36.251555    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz"]
Nov 03 00:29:36.252079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:36.251719    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz" podUID="48d61362-b777-4b66-84e2-03a542a6a761" containerName="payload-processing" containerID="cri-o://6c6042a62a30c313bd29fd60fcb879f43b3b8db79625256f4f02e031d31b0670" gracePeriod=30
Nov 03 00:29:36.273242 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:36.273209    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj"]
Nov 03 00:29:36.273765 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:36.273733    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj" podUID="b86d3c3e-36e1-406b-ae74-d8f846b60565" containerName="payload-pre-processing" containerID="cri-o://aecca86df24a981486d90a068b2c9061323f6ebd2d4e14314027838c0a42e7ef" gracePeriod=30
Nov 03 00:29:36.578365 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:36.577994    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j" podUID="29b49605-cfa9-464b-a91e-ec48f74cbef8" containerName="payload-processing" containerID="cri-o://c098c87fdabd6f2efb3a7a6b240f35ccdf36076c14dba6f75f8f88afe783f5ee" gracePeriod=25
Nov 03 00:29:36.731214 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:36.731184    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz"
Nov 03 00:29:36.796758 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:36.796727    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj"
Nov 03 00:29:36.857373 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:36.857339    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48d61362-b777-4b66-84e2-03a542a6a761-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48d61362-b777-4b66-84e2-03a542a6a761-plugins-config-volume\") pod \"48d61362-b777-4b66-84e2-03a542a6a761\" (UID: \"48d61362-b777-4b66-84e2-03a542a6a761\") "
Nov 03 00:29:36.857869 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:36.857613    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48d61362-b777-4b66-84e2-03a542a6a761-kube-api-access-jz4x6\" (UniqueName: \"kubernetes.io/projected/48d61362-b777-4b66-84e2-03a542a6a761-kube-api-access-jz4x6\") pod \"48d61362-b777-4b66-84e2-03a542a6a761\" (UID: \"48d61362-b777-4b66-84e2-03a542a6a761\") "
Nov 03 00:29:36.858952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:36.858224    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48d61362-b777-4b66-84e2-03a542a6a761-plugins-config-volume" pod "48d61362-b777-4b66-84e2-03a542a6a761" (UID: "48d61362-b777-4b66-84e2-03a542a6a761"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:29:36.861740 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:36.861177    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d61362-b777-4b66-84e2-03a542a6a761-kube-api-access-jz4x6" pod "48d61362-b777-4b66-84e2-03a542a6a761" (UID: "48d61362-b777-4b66-84e2-03a542a6a761"). InnerVolumeSpecName "kube-api-access-jz4x6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:29:36.958705 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:36.958675    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b86d3c3e-36e1-406b-ae74-d8f846b60565-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86d3c3e-36e1-406b-ae74-d8f846b60565-plugins-config-volume\") pod \"b86d3c3e-36e1-406b-ae74-d8f846b60565\" (UID: \"b86d3c3e-36e1-406b-ae74-d8f846b60565\") "
Nov 03 00:29:36.958888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:36.958820    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b86d3c3e-36e1-406b-ae74-d8f846b60565-kube-api-access-x7h9l\" (UniqueName: \"kubernetes.io/projected/b86d3c3e-36e1-406b-ae74-d8f846b60565-kube-api-access-x7h9l\") pod \"b86d3c3e-36e1-406b-ae74-d8f846b60565\" (UID: \"b86d3c3e-36e1-406b-ae74-d8f846b60565\") "
Nov 03 00:29:36.959171 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:36.959143    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86d3c3e-36e1-406b-ae74-d8f846b60565-plugins-config-volume" pod "b86d3c3e-36e1-406b-ae74-d8f846b60565" (UID: "b86d3c3e-36e1-406b-ae74-d8f846b60565"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:29:36.959486 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:36.959428    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz4x6\" (UniqueName: \"kubernetes.io/projected/48d61362-b777-4b66-84e2-03a542a6a761-kube-api-access-jz4x6\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:29:36.959486 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:36.959451    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48d61362-b777-4b66-84e2-03a542a6a761-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:29:36.959486 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:36.959461    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86d3c3e-36e1-406b-ae74-d8f846b60565-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:29:36.961138 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:36.961109    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86d3c3e-36e1-406b-ae74-d8f846b60565-kube-api-access-x7h9l" pod "b86d3c3e-36e1-406b-ae74-d8f846b60565" (UID: "b86d3c3e-36e1-406b-ae74-d8f846b60565"). InnerVolumeSpecName "kube-api-access-x7h9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:29:36.976434 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:36.976416    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j"
Nov 03 00:29:37.061039 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.060949    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29b49605-cfa9-464b-a91e-ec48f74cbef8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b49605-cfa9-464b-a91e-ec48f74cbef8-plugins-config-volume\") pod \"29b49605-cfa9-464b-a91e-ec48f74cbef8\" (UID: \"29b49605-cfa9-464b-a91e-ec48f74cbef8\") "
Nov 03 00:29:37.061213 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.061050    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b49605-cfa9-464b-a91e-ec48f74cbef8-kube-api-access-9tbzx\" (UniqueName: \"kubernetes.io/projected/29b49605-cfa9-464b-a91e-ec48f74cbef8-kube-api-access-9tbzx\") pod \"29b49605-cfa9-464b-a91e-ec48f74cbef8\" (UID: \"29b49605-cfa9-464b-a91e-ec48f74cbef8\") "
Nov 03 00:29:37.061431 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.061399    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29b49605-cfa9-464b-a91e-ec48f74cbef8-plugins-config-volume" pod "29b49605-cfa9-464b-a91e-ec48f74cbef8" (UID: "29b49605-cfa9-464b-a91e-ec48f74cbef8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:29:37.061686 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.061665    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b49605-cfa9-464b-a91e-ec48f74cbef8-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:29:37.061845 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.061821    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7h9l\" (UniqueName: \"kubernetes.io/projected/b86d3c3e-36e1-406b-ae74-d8f846b60565-kube-api-access-x7h9l\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:29:37.067050 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.067020    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b49605-cfa9-464b-a91e-ec48f74cbef8-kube-api-access-9tbzx" pod "29b49605-cfa9-464b-a91e-ec48f74cbef8" (UID: "29b49605-cfa9-464b-a91e-ec48f74cbef8"). InnerVolumeSpecName "kube-api-access-9tbzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:29:37.163128 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.163095    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tbzx\" (UniqueName: \"kubernetes.io/projected/29b49605-cfa9-464b-a91e-ec48f74cbef8-kube-api-access-9tbzx\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:29:37.323956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.323862    5012 generic.go:356] "Generic (PLEG): container finished" podID="b86d3c3e-36e1-406b-ae74-d8f846b60565" containerID="aecca86df24a981486d90a068b2c9061323f6ebd2d4e14314027838c0a42e7ef" exitCode=0
Nov 03 00:29:37.323956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.323944    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj"
Nov 03 00:29:37.324397 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.323944    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj" event={"ID":"b86d3c3e-36e1-406b-ae74-d8f846b60565","Type":"ContainerDied","Data":"aecca86df24a981486d90a068b2c9061323f6ebd2d4e14314027838c0a42e7ef"}
Nov 03 00:29:37.324397 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.324068    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj" event={"ID":"b86d3c3e-36e1-406b-ae74-d8f846b60565","Type":"ContainerDied","Data":"c87e2ec241963bf794649721b00ea25cb3d62007495971ee5782158d03633841"}
Nov 03 00:29:37.324397 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.324091    5012 scope.go:122] "RemoveContainer" containerID="aecca86df24a981486d90a068b2c9061323f6ebd2d4e14314027838c0a42e7ef"
Nov 03 00:29:37.325783 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.325744    5012 generic.go:356] "Generic (PLEG): container finished" podID="48d61362-b777-4b66-84e2-03a542a6a761" containerID="6c6042a62a30c313bd29fd60fcb879f43b3b8db79625256f4f02e031d31b0670" exitCode=0
Nov 03 00:29:37.325899 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.325808    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz"
Nov 03 00:29:37.325899 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.325829    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz" event={"ID":"48d61362-b777-4b66-84e2-03a542a6a761","Type":"ContainerDied","Data":"6c6042a62a30c313bd29fd60fcb879f43b3b8db79625256f4f02e031d31b0670"}
Nov 03 00:29:37.326034 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.325966    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz" event={"ID":"48d61362-b777-4b66-84e2-03a542a6a761","Type":"ContainerDied","Data":"df7926fb6e9c66da78682f9f21b64f234e6a6c36684d908c67d4bdc8bddd477f"}
Nov 03 00:29:37.327791 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.327768    5012 generic.go:356] "Generic (PLEG): container finished" podID="29b49605-cfa9-464b-a91e-ec48f74cbef8" containerID="c098c87fdabd6f2efb3a7a6b240f35ccdf36076c14dba6f75f8f88afe783f5ee" exitCode=0
Nov 03 00:29:37.327903 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.327823    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j"
Nov 03 00:29:37.327903 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.327834    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j" event={"ID":"29b49605-cfa9-464b-a91e-ec48f74cbef8","Type":"ContainerDied","Data":"c098c87fdabd6f2efb3a7a6b240f35ccdf36076c14dba6f75f8f88afe783f5ee"}
Nov 03 00:29:37.327903 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.327890    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j" event={"ID":"29b49605-cfa9-464b-a91e-ec48f74cbef8","Type":"ContainerDied","Data":"c0076a0f88894185b289fbbd78e9d59769b043fad181fb42654480d012476a0c"}
Nov 03 00:29:37.336252 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.336224    5012 scope.go:122] "RemoveContainer" containerID="aecca86df24a981486d90a068b2c9061323f6ebd2d4e14314027838c0a42e7ef"
Nov 03 00:29:37.336517 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:29:37.336493    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aecca86df24a981486d90a068b2c9061323f6ebd2d4e14314027838c0a42e7ef\": container with ID starting with aecca86df24a981486d90a068b2c9061323f6ebd2d4e14314027838c0a42e7ef not found: ID does not exist" containerID="aecca86df24a981486d90a068b2c9061323f6ebd2d4e14314027838c0a42e7ef"
Nov 03 00:29:37.336600 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.336525    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aecca86df24a981486d90a068b2c9061323f6ebd2d4e14314027838c0a42e7ef"} err="failed to get container status \"aecca86df24a981486d90a068b2c9061323f6ebd2d4e14314027838c0a42e7ef\": rpc error: code = NotFound desc = could not find container \"aecca86df24a981486d90a068b2c9061323f6ebd2d4e14314027838c0a42e7ef\": container with ID starting with aecca86df24a981486d90a068b2c9061323f6ebd2d4e14314027838c0a42e7ef not found: ID does not exist"
Nov 03 00:29:37.336600 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.336547    5012 scope.go:122] "RemoveContainer" containerID="6c6042a62a30c313bd29fd60fcb879f43b3b8db79625256f4f02e031d31b0670"
Nov 03 00:29:37.346076 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.346058    5012 scope.go:122] "RemoveContainer" containerID="6c6042a62a30c313bd29fd60fcb879f43b3b8db79625256f4f02e031d31b0670"
Nov 03 00:29:37.346381 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:29:37.346362    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c6042a62a30c313bd29fd60fcb879f43b3b8db79625256f4f02e031d31b0670\": container with ID starting with 6c6042a62a30c313bd29fd60fcb879f43b3b8db79625256f4f02e031d31b0670 not found: ID does not exist" containerID="6c6042a62a30c313bd29fd60fcb879f43b3b8db79625256f4f02e031d31b0670"
Nov 03 00:29:37.346477 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.346387    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c6042a62a30c313bd29fd60fcb879f43b3b8db79625256f4f02e031d31b0670"} err="failed to get container status \"6c6042a62a30c313bd29fd60fcb879f43b3b8db79625256f4f02e031d31b0670\": rpc error: code = NotFound desc = could not find container \"6c6042a62a30c313bd29fd60fcb879f43b3b8db79625256f4f02e031d31b0670\": container with ID starting with 6c6042a62a30c313bd29fd60fcb879f43b3b8db79625256f4f02e031d31b0670 not found: ID does not exist"
Nov 03 00:29:37.346477 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.346404    5012 scope.go:122] "RemoveContainer" containerID="c098c87fdabd6f2efb3a7a6b240f35ccdf36076c14dba6f75f8f88afe783f5ee"
Nov 03 00:29:37.356139 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.356075    5012 scope.go:122] "RemoveContainer" containerID="c098c87fdabd6f2efb3a7a6b240f35ccdf36076c14dba6f75f8f88afe783f5ee"
Nov 03 00:29:37.356536 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:29:37.356515    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c098c87fdabd6f2efb3a7a6b240f35ccdf36076c14dba6f75f8f88afe783f5ee\": container with ID starting with c098c87fdabd6f2efb3a7a6b240f35ccdf36076c14dba6f75f8f88afe783f5ee not found: ID does not exist" containerID="c098c87fdabd6f2efb3a7a6b240f35ccdf36076c14dba6f75f8f88afe783f5ee"
Nov 03 00:29:37.356625 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.356541    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c098c87fdabd6f2efb3a7a6b240f35ccdf36076c14dba6f75f8f88afe783f5ee"} err="failed to get container status \"c098c87fdabd6f2efb3a7a6b240f35ccdf36076c14dba6f75f8f88afe783f5ee\": rpc error: code = NotFound desc = could not find container \"c098c87fdabd6f2efb3a7a6b240f35ccdf36076c14dba6f75f8f88afe783f5ee\": container with ID starting with c098c87fdabd6f2efb3a7a6b240f35ccdf36076c14dba6f75f8f88afe783f5ee not found: ID does not exist"
Nov 03 00:29:37.363883 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:29:37.363860    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:29:37.372396 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.372373    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j"]
Nov 03 00:29:37.382481 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.382460    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-846492-5b87b8fc7f-46q7j"]
Nov 03 00:29:37.406496 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.406478    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz"]
Nov 03 00:29:37.410005 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.409985    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e37c9d-5d9569d6db-4bkhz"]
Nov 03 00:29:37.438931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.438889    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj"]
Nov 03 00:29:37.445251 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:37.445220    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e37c9d-6dbbd4494c-49jkj"]
Nov 03 00:29:39.245069 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:39.245032    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29b49605-cfa9-464b-a91e-ec48f74cbef8" path="/var/lib/kubelet/pods/29b49605-cfa9-464b-a91e-ec48f74cbef8/volumes"
Nov 03 00:29:39.245470 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:39.245455    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48d61362-b777-4b66-84e2-03a542a6a761" path="/var/lib/kubelet/pods/48d61362-b777-4b66-84e2-03a542a6a761/volumes"
Nov 03 00:29:39.245836 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:39.245824    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b86d3c3e-36e1-406b-ae74-d8f846b60565" path="/var/lib/kubelet/pods/b86d3c3e-36e1-406b-ae74-d8f846b60565/volumes"
Nov 03 00:29:40.409284 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.409255    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"]
Nov 03 00:29:40.409952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.409930    5012 cpu_manager.go:414] "containerMap: removing container" podUID="b86d3c3e-36e1-406b-ae74-d8f846b60565" containerName="payload-pre-processing"
Nov 03 00:29:40.409952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.409948    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b86d3c3e-36e1-406b-ae74-d8f846b60565" containerName="payload-pre-processing"
Nov 03 00:29:40.410156 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.409965    5012 cpu_manager.go:414] "containerMap: removing container" podUID="48d61362-b777-4b66-84e2-03a542a6a761" containerName="payload-processing"
Nov 03 00:29:40.410156 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.409973    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48d61362-b777-4b66-84e2-03a542a6a761" containerName="payload-processing"
Nov 03 00:29:40.410156 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.410009    5012 cpu_manager.go:414] "containerMap: removing container" podUID="29b49605-cfa9-464b-a91e-ec48f74cbef8" containerName="payload-processing"
Nov 03 00:29:40.410156 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.410016    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29b49605-cfa9-464b-a91e-ec48f74cbef8" containerName="payload-processing"
Nov 03 00:29:40.410351 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.410229    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="b86d3c3e-36e1-406b-ae74-d8f846b60565" containerName="payload-pre-processing"
Nov 03 00:29:40.410351 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.410244    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="48d61362-b777-4b66-84e2-03a542a6a761" containerName="payload-processing"
Nov 03 00:29:40.410351 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.410257    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="29b49605-cfa9-464b-a91e-ec48f74cbef8" containerName="payload-processing"
Nov 03 00:29:40.415586 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.415566    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.422376 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.422347    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-data\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.422535 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.422395    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r85z\" (UniqueName: \"kubernetes.io/projected/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-kube-api-access-7r85z\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.422535 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.422451    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-token\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.422535 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.422494    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-workload-socket\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.422731 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.422553    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-envoy\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.422731 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.422583    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-credential-socket\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.422731 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.422642    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-workload-certs\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.422958 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.422780    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istiod-ca-cert\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.422958 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.422813    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-podinfo\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.436457 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.436430    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"]
Nov 03 00:29:40.441215 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.441197    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ea275f-openshift-default-dockercfg-rhh2d\""
Nov 03 00:29:40.523752 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.523709    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-workload-certs\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.523981 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.523802    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istiod-ca-cert\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.523981 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.523827    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-podinfo\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.524141 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.524006    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-data\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.524141 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.524044    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r85z\" (UniqueName: \"kubernetes.io/projected/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-kube-api-access-7r85z\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.524141 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.524129    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-workload-certs\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.524303 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.524090    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-token\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.524303 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.524281    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-workload-socket\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.524405 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.524346    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-envoy\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.524405 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.524375    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-credential-socket\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.524523 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.524378    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-data\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.524868 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.524818    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-credential-socket\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.524868 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.524951    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-workload-socket\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.525373 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.525169    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istiod-ca-cert\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.526923 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.526881    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-envoy\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.527055 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.527032    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-podinfo\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.570958 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.551557    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-token\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.596738 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.596654    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r85z\" (UniqueName: \"kubernetes.io/projected/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-kube-api-access-7r85z\") pod \"e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") " pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:40.732466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:40.732383    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:41.082877 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:29:41.082848    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b16caf_d83f_4c7b_a1fc_bdcb7d061653.slice/crio-fdaaea1dfc462ef1d2559cc9e430b3dc8ed607b8d6246addbaef10f8cd167257 WatchSource:0}: Error finding container fdaaea1dfc462ef1d2559cc9e430b3dc8ed607b8d6246addbaef10f8cd167257: Status 404 returned error can't find the container with id fdaaea1dfc462ef1d2559cc9e430b3dc8ed607b8d6246addbaef10f8cd167257
Nov 03 00:29:41.083461 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:41.083386    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"]
Nov 03 00:29:41.085342 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:41.085314    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:29:41.085416 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:41.085378    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:29:41.085416 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:41.085408    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:29:41.365006 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:41.364967    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg" event={"ID":"90b16caf-d83f-4c7b-a1fc-bdcb7d061653","Type":"ContainerStarted","Data":"fd9b8edd5fd21bf21e78c6917f58e7c47fe4713f1a392f53830351f7ca1c6a7e"}
Nov 03 00:29:41.365203 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:41.365032    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg" event={"ID":"90b16caf-d83f-4c7b-a1fc-bdcb7d061653","Type":"ContainerStarted","Data":"fdaaea1dfc462ef1d2559cc9e430b3dc8ed607b8d6246addbaef10f8cd167257"}
Nov 03 00:29:41.403275 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:41.403222    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg" podStartSLOduration=1.40320687 podStartE2EDuration="1.40320687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:41.400872891 +0000 UTC m=+1464.280456935" watchObservedRunningTime="2026-11-03 00:29:41.40320687 +0000 UTC m=+1464.282790913"
Nov 03 00:29:41.733085 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:41.733007    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:42.738920 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:42.738877    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:43.381170 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:43.381140    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:43.382004 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:43.381986    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:29:46.404351 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:46.404312    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"]
Nov 03 00:29:46.404847 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:46.404548    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms" podUID="ba93722b-9169-4628-9f61-baaef18a68a7" containerName="istio-proxy" containerID="cri-o://5c497e9f8aff8800645c483955230201067624b6ddc3e390faa03d406767d944" gracePeriod=30
Nov 03 00:29:46.536248 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:46.536212    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t"]
Nov 03 00:29:46.541087 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:46.541064    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t"
Nov 03 00:29:46.580069 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:46.580039    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ea275f-dockercfg-bcl9x\""
Nov 03 00:29:46.593297 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:46.593230    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t"]
Nov 03 00:29:46.676981 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:46.634565    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ea275f\""
Nov 03 00:29:46.738780 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:46.738743    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kjtq\" (UniqueName: \"kubernetes.io/projected/09c04970-f7a0-401e-a077-79a5012fb572-kube-api-access-6kjtq\") pod \"payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t\" (UID: \"09c04970-f7a0-401e-a077-79a5012fb572\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t"
Nov 03 00:29:46.739031 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:46.738974    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09c04970-f7a0-401e-a077-79a5012fb572-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t\" (UID: \"09c04970-f7a0-401e-a077-79a5012fb572\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t"
Nov 03 00:29:46.841750 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:46.841715    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09c04970-f7a0-401e-a077-79a5012fb572-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t\" (UID: \"09c04970-f7a0-401e-a077-79a5012fb572\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t"
Nov 03 00:29:46.842266 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:46.842239    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kjtq\" (UniqueName: \"kubernetes.io/projected/09c04970-f7a0-401e-a077-79a5012fb572-kube-api-access-6kjtq\") pod \"payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t\" (UID: \"09c04970-f7a0-401e-a077-79a5012fb572\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t"
Nov 03 00:29:46.843440 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:46.843417    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09c04970-f7a0-401e-a077-79a5012fb572-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t\" (UID: \"09c04970-f7a0-401e-a077-79a5012fb572\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t"
Nov 03 00:29:46.861583 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:46.861564    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kjtq\" (UniqueName: \"kubernetes.io/projected/09c04970-f7a0-401e-a077-79a5012fb572-kube-api-access-6kjtq\") pod \"payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t\" (UID: \"09c04970-f7a0-401e-a077-79a5012fb572\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t"
Nov 03 00:29:47.156262 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.156230    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t"
Nov 03 00:29:47.279256 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.279187    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff"]
Nov 03 00:29:47.286696 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.286671    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff"
Nov 03 00:29:47.300763 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.300621    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-846492\""
Nov 03 00:29:47.300763 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.300731    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-846492-dockercfg-vfhf5\""
Nov 03 00:29:47.315590 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.315557    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff"]
Nov 03 00:29:47.351090 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.351050    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72825023-1f03-47a3-a213-4cf1656e108a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff\" (UID: \"72825023-1f03-47a3-a213-4cf1656e108a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff"
Nov 03 00:29:47.351285 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.351156    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdhbv\" (UniqueName: \"kubernetes.io/projected/72825023-1f03-47a3-a213-4cf1656e108a-kube-api-access-xdhbv\") pod \"payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff\" (UID: \"72825023-1f03-47a3-a213-4cf1656e108a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff"
Nov 03 00:29:47.452702 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.452613    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72825023-1f03-47a3-a213-4cf1656e108a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff\" (UID: \"72825023-1f03-47a3-a213-4cf1656e108a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff"
Nov 03 00:29:47.453234 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.452719    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdhbv\" (UniqueName: \"kubernetes.io/projected/72825023-1f03-47a3-a213-4cf1656e108a-kube-api-access-xdhbv\") pod \"payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff\" (UID: \"72825023-1f03-47a3-a213-4cf1656e108a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff"
Nov 03 00:29:47.453750 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.453723    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72825023-1f03-47a3-a213-4cf1656e108a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff\" (UID: \"72825023-1f03-47a3-a213-4cf1656e108a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff"
Nov 03 00:29:47.472793 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.472762    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdhbv\" (UniqueName: \"kubernetes.io/projected/72825023-1f03-47a3-a213-4cf1656e108a-kube-api-access-xdhbv\") pod \"payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff\" (UID: \"72825023-1f03-47a3-a213-4cf1656e108a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff"
Nov 03 00:29:47.481056 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.481027    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t"]
Nov 03 00:29:47.482505 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:29:47.482474    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c04970_f7a0_401e_a077_79a5012fb572.slice/crio-09e55772a81a21ca539a121a0605c2a3d93275256927a2e4c2503177066438ae WatchSource:0}: Error finding container 09e55772a81a21ca539a121a0605c2a3d93275256927a2e4c2503177066438ae: Status 404 returned error can't find the container with id 09e55772a81a21ca539a121a0605c2a3d93275256927a2e4c2503177066438ae
Nov 03 00:29:47.516450 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.516335    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"]
Nov 03 00:29:47.523023 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.523003    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.546966 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.546295    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-191362-openshift-default-dockercfg-4rnzz\""
Nov 03 00:29:47.555744 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.555709    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5138f419-8f8a-4402-b134-8fdc698d5de5-istiod-ca-cert\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.555923 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.555760    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-credential-socket\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.555923 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.555822    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-envoy\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.555923 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.555858    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"]
Nov 03 00:29:47.556121 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.555936    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-podinfo\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.556121 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.556031    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lts5r\" (UniqueName: \"kubernetes.io/projected/5138f419-8f8a-4402-b134-8fdc698d5de5-kube-api-access-lts5r\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.556289 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.556119    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-data\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.556289 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.556267    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-workload-certs\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.556395 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.556363    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-token\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.556458 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.556421    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-workload-socket\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.606200 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.606177    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff"
Nov 03 00:29:47.657471 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.657421    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-data\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.657620 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.657501    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-workload-certs\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.657620 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.657549    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-token\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.657620 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.657573    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-workload-socket\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.657620 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.657601    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5138f419-8f8a-4402-b134-8fdc698d5de5-istiod-ca-cert\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.657620 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.657621    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-credential-socket\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.657789 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.657661    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-envoy\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.657789 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.657679    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-podinfo\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.657789 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.657722    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lts5r\" (UniqueName: \"kubernetes.io/projected/5138f419-8f8a-4402-b134-8fdc698d5de5-kube-api-access-lts5r\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.658992 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.658208    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-workload-socket\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.658992 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.658388    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-data\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.658992 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.658548    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5138f419-8f8a-4402-b134-8fdc698d5de5-istiod-ca-cert\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.658992 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.658617    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-workload-certs\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.658992 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.658751    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-credential-socket\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.662166 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.662141    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-envoy\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.663097 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.662904    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-podinfo\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.679423 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.679372    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lts5r\" (UniqueName: \"kubernetes.io/projected/5138f419-8f8a-4402-b134-8fdc698d5de5-kube-api-access-lts5r\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.680773 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.680747    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm"]
Nov 03 00:29:47.683709 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.683682    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-token\") pod \"e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") " pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.687230 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.687208    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm"
Nov 03 00:29:47.700057 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.700038    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm"]
Nov 03 00:29:47.761999 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.761279    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2wfw\" (UniqueName: \"kubernetes.io/projected/0528abca-dac8-491c-b6ab-d183adf6bf8a-kube-api-access-t2wfw\") pod \"payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm\" (UID: \"0528abca-dac8-491c-b6ab-d183adf6bf8a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm"
Nov 03 00:29:47.761999 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.761442    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0528abca-dac8-491c-b6ab-d183adf6bf8a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm\" (UID: \"0528abca-dac8-491c-b6ab-d183adf6bf8a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm"
Nov 03 00:29:47.856464 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.856170    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:47.868324 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.864841    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0528abca-dac8-491c-b6ab-d183adf6bf8a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm\" (UID: \"0528abca-dac8-491c-b6ab-d183adf6bf8a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm"
Nov 03 00:29:47.868324 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.865132    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2wfw\" (UniqueName: \"kubernetes.io/projected/0528abca-dac8-491c-b6ab-d183adf6bf8a-kube-api-access-t2wfw\") pod \"payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm\" (UID: \"0528abca-dac8-491c-b6ab-d183adf6bf8a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm"
Nov 03 00:29:47.868324 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.866390    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0528abca-dac8-491c-b6ab-d183adf6bf8a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm\" (UID: \"0528abca-dac8-491c-b6ab-d183adf6bf8a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm"
Nov 03 00:29:47.910564 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.910502    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2wfw\" (UniqueName: \"kubernetes.io/projected/0528abca-dac8-491c-b6ab-d183adf6bf8a-kube-api-access-t2wfw\") pod \"payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm\" (UID: \"0528abca-dac8-491c-b6ab-d183adf6bf8a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm"
Nov 03 00:29:47.938192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:47.938161    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff"]
Nov 03 00:29:47.945104 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:29:47.945075    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72825023_1f03_47a3_a213_4cf1656e108a.slice/crio-67cb46bc2550fb495d18fed80717b418a38d05fca6d024957860140c44ce596d WatchSource:0}: Error finding container 67cb46bc2550fb495d18fed80717b418a38d05fca6d024957860140c44ce596d: Status 404 returned error can't find the container with id 67cb46bc2550fb495d18fed80717b418a38d05fca6d024957860140c44ce596d
Nov 03 00:29:48.006278 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:48.006245    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm"
Nov 03 00:29:48.264119 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:29:48.264091    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5138f419_8f8a_4402_b134_8fdc698d5de5.slice/crio-79dfd057f40fa8f0776af1d456e82134dc40f9946ffbb2d359d96f05a702ccf0 WatchSource:0}: Error finding container 79dfd057f40fa8f0776af1d456e82134dc40f9946ffbb2d359d96f05a702ccf0: Status 404 returned error can't find the container with id 79dfd057f40fa8f0776af1d456e82134dc40f9946ffbb2d359d96f05a702ccf0
Nov 03 00:29:48.264293 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:48.264268    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"]
Nov 03 00:29:48.266776 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:48.266743    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:29:48.266871 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:48.266813    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:29:48.266871 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:48.266843    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:29:48.336271 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:48.336236    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm"]
Nov 03 00:29:48.339723 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:29:48.339698    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0528abca_dac8_491c_b6ab_d183adf6bf8a.slice/crio-52ebed9f43315f3ab69c205e6f3f85be6c61441d8bc47a1da8e69824a550e5d1 WatchSource:0}: Error finding container 52ebed9f43315f3ab69c205e6f3f85be6c61441d8bc47a1da8e69824a550e5d1: Status 404 returned error can't find the container with id 52ebed9f43315f3ab69c205e6f3f85be6c61441d8bc47a1da8e69824a550e5d1
Nov 03 00:29:48.443965 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:48.442753    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm" event={"ID":"0528abca-dac8-491c-b6ab-d183adf6bf8a","Type":"ContainerStarted","Data":"52ebed9f43315f3ab69c205e6f3f85be6c61441d8bc47a1da8e69824a550e5d1"}
Nov 03 00:29:48.448932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:48.444749    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff" event={"ID":"72825023-1f03-47a3-a213-4cf1656e108a","Type":"ContainerStarted","Data":"edabaaa2ef1bd3c814da594d5207af5384aa6248be2e71ab3f96b7e0b4ee095b"}
Nov 03 00:29:48.448932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:48.444801    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff" event={"ID":"72825023-1f03-47a3-a213-4cf1656e108a","Type":"ContainerStarted","Data":"67cb46bc2550fb495d18fed80717b418a38d05fca6d024957860140c44ce596d"}
Nov 03 00:29:48.448932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:48.445382    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff"
Nov 03 00:29:48.448932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:48.447789    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b" event={"ID":"5138f419-8f8a-4402-b134-8fdc698d5de5","Type":"ContainerStarted","Data":"2e35845ac19b31c9a771449c8d3bacefb3048e0579961b3e09315714544715cc"}
Nov 03 00:29:48.448932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:48.447830    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b" event={"ID":"5138f419-8f8a-4402-b134-8fdc698d5de5","Type":"ContainerStarted","Data":"79dfd057f40fa8f0776af1d456e82134dc40f9946ffbb2d359d96f05a702ccf0"}
Nov 03 00:29:48.451974 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:48.450392    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t" event={"ID":"09c04970-f7a0-401e-a077-79a5012fb572","Type":"ContainerStarted","Data":"6057f25729ea95d92f9080a9d1c3924ee893f853e2bcd4d8cf502e76570e12df"}
Nov 03 00:29:48.451974 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:48.450435    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t" event={"ID":"09c04970-f7a0-401e-a077-79a5012fb572","Type":"ContainerStarted","Data":"09e55772a81a21ca539a121a0605c2a3d93275256927a2e4c2503177066438ae"}
Nov 03 00:29:48.451974 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:48.451114    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t"
Nov 03 00:29:48.501270 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:48.501208    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff" podStartSLOduration=1.501189953 podStartE2EDuration="1.501189953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:48.488855716 +0000 UTC m=+1471.368439764" watchObservedRunningTime="2026-11-03 00:29:48.501189953 +0000 UTC m=+1471.380773999"
Nov 03 00:29:48.528962 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:48.528857    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t" podStartSLOduration=2.52884214 podStartE2EDuration="2.52884214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:48.526450544 +0000 UTC m=+1471.406034583" watchObservedRunningTime="2026-11-03 00:29:48.52884214 +0000 UTC m=+1471.408426183"
Nov 03 00:29:48.567510 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:48.567455    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b" podStartSLOduration=1.5674433730000001 podStartE2EDuration="1.567443373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:48.563404016 +0000 UTC m=+1471.442988058" watchObservedRunningTime="2026-11-03 00:29:48.567443373 +0000 UTC m=+1471.447027416"
Nov 03 00:29:48.857519 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:48.857486    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:49.460764 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:49.460722    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm" event={"ID":"0528abca-dac8-491c-b6ab-d183adf6bf8a","Type":"ContainerStarted","Data":"6aefa2f8e7e7b12322d1396a7f667ed32325159a525042217deb7b202581b301"}
Nov 03 00:29:49.460979 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:49.460822    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm"
Nov 03 00:29:49.500582 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:49.500525    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm" podStartSLOduration=2.500512743 podStartE2EDuration="2.500512743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:49.497667493 +0000 UTC m=+1472.377251537" watchObservedRunningTime="2026-11-03 00:29:49.500512743 +0000 UTC m=+1472.380096787"
Nov 03 00:29:49.874929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:49.870165    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:50.468803 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:50.468759    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:50.470127 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:50.470098    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:29:51.779935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:51.778478    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:29:51.941356 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:51.941266    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-workload-certs\") pod \"ba93722b-9169-4628-9f61-baaef18a68a7\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") "
Nov 03 00:29:51.941356 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:51.941309    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba93722b-9169-4628-9f61-baaef18a68a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba93722b-9169-4628-9f61-baaef18a68a7-istio-podinfo\") pod \"ba93722b-9169-4628-9f61-baaef18a68a7\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") "
Nov 03 00:29:51.941356 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:51.941331    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-credential-socket\") pod \"ba93722b-9169-4628-9f61-baaef18a68a7\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") "
Nov 03 00:29:51.941662 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:51.941379    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-workload-socket\") pod \"ba93722b-9169-4628-9f61-baaef18a68a7\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") "
Nov 03 00:29:51.941662 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:51.941466    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba93722b-9169-4628-9f61-baaef18a68a7-kube-api-access-6kc22\" (UniqueName: \"kubernetes.io/projected/ba93722b-9169-4628-9f61-baaef18a68a7-kube-api-access-6kc22\") pod \"ba93722b-9169-4628-9f61-baaef18a68a7\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") "
Nov 03 00:29:51.941662 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:51.941507    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-istio-data\") pod \"ba93722b-9169-4628-9f61-baaef18a68a7\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") "
Nov 03 00:29:51.941662 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:51.941544    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba93722b-9169-4628-9f61-baaef18a68a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba93722b-9169-4628-9f61-baaef18a68a7-istiod-ca-cert\") pod \"ba93722b-9169-4628-9f61-baaef18a68a7\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") "
Nov 03 00:29:51.941888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:51.941670    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-istio-envoy\") pod \"ba93722b-9169-4628-9f61-baaef18a68a7\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") "
Nov 03 00:29:51.941888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:51.941756    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba93722b-9169-4628-9f61-baaef18a68a7-istio-token\" (UniqueName: \"kubernetes.io/projected/ba93722b-9169-4628-9f61-baaef18a68a7-istio-token\") pod \"ba93722b-9169-4628-9f61-baaef18a68a7\" (UID: \"ba93722b-9169-4628-9f61-baaef18a68a7\") "
Nov 03 00:29:51.941888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:51.941769    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-istio-data" pod "ba93722b-9169-4628-9f61-baaef18a68a7" (UID: "ba93722b-9169-4628-9f61-baaef18a68a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:29:51.941888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:51.941874    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-workload-socket" pod "ba93722b-9169-4628-9f61-baaef18a68a7" (UID: "ba93722b-9169-4628-9f61-baaef18a68a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:29:51.942172 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:51.942006    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba93722b-9169-4628-9f61-baaef18a68a7-istiod-ca-cert" pod "ba93722b-9169-4628-9f61-baaef18a68a7" (UID: "ba93722b-9169-4628-9f61-baaef18a68a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:29:51.942172 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:51.941612    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-workload-certs" pod "ba93722b-9169-4628-9f61-baaef18a68a7" (UID: "ba93722b-9169-4628-9f61-baaef18a68a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:29:51.942669 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:51.942584    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:29:51.942669 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:51.942613    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba93722b-9169-4628-9f61-baaef18a68a7-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:29:51.942669 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:51.942628    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:29:51.942669 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:51.942640    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:29:51.942991 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:51.942766    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-credential-socket" pod "ba93722b-9169-4628-9f61-baaef18a68a7" (UID: "ba93722b-9169-4628-9f61-baaef18a68a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:29:51.944043 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:51.944019    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba93722b-9169-4628-9f61-baaef18a68a7-istio-podinfo" pod "ba93722b-9169-4628-9f61-baaef18a68a7" (UID: "ba93722b-9169-4628-9f61-baaef18a68a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:29:51.944156 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:51.944137    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-istio-envoy" pod "ba93722b-9169-4628-9f61-baaef18a68a7" (UID: "ba93722b-9169-4628-9f61-baaef18a68a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:29:51.944548 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:51.944523    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba93722b-9169-4628-9f61-baaef18a68a7-kube-api-access-6kc22" pod "ba93722b-9169-4628-9f61-baaef18a68a7" (UID: "ba93722b-9169-4628-9f61-baaef18a68a7"). InnerVolumeSpecName "kube-api-access-6kc22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:29:51.944630 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:51.944605    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba93722b-9169-4628-9f61-baaef18a68a7-istio-token" pod "ba93722b-9169-4628-9f61-baaef18a68a7" (UID: "ba93722b-9169-4628-9f61-baaef18a68a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:29:52.034229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:52.034189    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ef32911-k2gjm"]
Nov 03 00:29:52.044094 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:52.044046    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba93722b-9169-4628-9f61-baaef18a68a7-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:29:52.044314 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:52.044287    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba93722b-9169-4628-9f61-baaef18a68a7-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:29:52.044416 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:52.044317    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:29:52.044416 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:52.044333    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kc22\" (UniqueName: \"kubernetes.io/projected/ba93722b-9169-4628-9f61-baaef18a68a7-kube-api-access-6kc22\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:29:52.044416 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:52.044348    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba93722b-9169-4628-9f61-baaef18a68a7-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:29:52.044903 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:52.044857    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ef32911-k2gjm"]
Nov 03 00:29:52.521322 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:52.519386    5012 generic.go:356] "Generic (PLEG): container finished" podID="ba93722b-9169-4628-9f61-baaef18a68a7" containerID="5c497e9f8aff8800645c483955230201067624b6ddc3e390faa03d406767d944" exitCode=0
Nov 03 00:29:52.521322 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:52.519460    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"
Nov 03 00:29:52.521322 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:52.519514    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms" event={"ID":"ba93722b-9169-4628-9f61-baaef18a68a7","Type":"ContainerDied","Data":"5c497e9f8aff8800645c483955230201067624b6ddc3e390faa03d406767d944"}
Nov 03 00:29:52.521322 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:52.519593    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms" event={"ID":"ba93722b-9169-4628-9f61-baaef18a68a7","Type":"ContainerDied","Data":"792161c75b5b7fb1bb5f40187cd36564f2794054028c33b19d4620d7560455c7"}
Nov 03 00:29:52.521322 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:52.519631    5012 scope.go:122] "RemoveContainer" containerID="5c497e9f8aff8800645c483955230201067624b6ddc3e390faa03d406767d944"
Nov 03 00:29:52.532972 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:52.532948    5012 scope.go:122] "RemoveContainer" containerID="f4878d8508f1d06d42d201a2f1dcd80e113ab7824553830502fc5ce404ce1390"
Nov 03 00:29:52.544819 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:52.544792    5012 scope.go:122] "RemoveContainer" containerID="5c497e9f8aff8800645c483955230201067624b6ddc3e390faa03d406767d944"
Nov 03 00:29:52.545147 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:29:52.545118    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c497e9f8aff8800645c483955230201067624b6ddc3e390faa03d406767d944\": container with ID starting with 5c497e9f8aff8800645c483955230201067624b6ddc3e390faa03d406767d944 not found: ID does not exist" containerID="5c497e9f8aff8800645c483955230201067624b6ddc3e390faa03d406767d944"
Nov 03 00:29:52.545249 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:52.545159    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c497e9f8aff8800645c483955230201067624b6ddc3e390faa03d406767d944"} err="failed to get container status \"5c497e9f8aff8800645c483955230201067624b6ddc3e390faa03d406767d944\": rpc error: code = NotFound desc = could not find container \"5c497e9f8aff8800645c483955230201067624b6ddc3e390faa03d406767d944\": container with ID starting with 5c497e9f8aff8800645c483955230201067624b6ddc3e390faa03d406767d944 not found: ID does not exist"
Nov 03 00:29:52.545249 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:52.545187    5012 scope.go:122] "RemoveContainer" containerID="f4878d8508f1d06d42d201a2f1dcd80e113ab7824553830502fc5ce404ce1390"
Nov 03 00:29:52.545491 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:29:52.545472    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4878d8508f1d06d42d201a2f1dcd80e113ab7824553830502fc5ce404ce1390\": container with ID starting with f4878d8508f1d06d42d201a2f1dcd80e113ab7824553830502fc5ce404ce1390 not found: ID does not exist" containerID="f4878d8508f1d06d42d201a2f1dcd80e113ab7824553830502fc5ce404ce1390"
Nov 03 00:29:52.545571 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:52.545498    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4878d8508f1d06d42d201a2f1dcd80e113ab7824553830502fc5ce404ce1390"} err="failed to get container status \"f4878d8508f1d06d42d201a2f1dcd80e113ab7824553830502fc5ce404ce1390\": rpc error: code = NotFound desc = could not find container \"f4878d8508f1d06d42d201a2f1dcd80e113ab7824553830502fc5ce404ce1390\": container with ID starting with f4878d8508f1d06d42d201a2f1dcd80e113ab7824553830502fc5ce404ce1390 not found: ID does not exist"
Nov 03 00:29:52.581272 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:52.581238    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"]
Nov 03 00:29:52.594005 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:52.593972    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e37c9d-openshift-default-6478988df5-khqms"]
Nov 03 00:29:53.265053 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:53.262875    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed" path="/var/lib/kubelet/pods/3c5bb4d3-7aa8-4213-be5c-2cf52eecbfed/volumes"
Nov 03 00:29:53.265053 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:53.263825    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba93722b-9169-4628-9f61-baaef18a68a7" path="/var/lib/kubelet/pods/ba93722b-9169-4628-9f61-baaef18a68a7/volumes"
Nov 03 00:29:55.935005 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:55.934970    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc"]
Nov 03 00:29:55.935639 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:55.935626    5012 cpu_manager.go:414] "containerMap: removing container" podUID="ba93722b-9169-4628-9f61-baaef18a68a7" containerName="istio-proxy"
Nov 03 00:29:55.935687 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:55.935640    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba93722b-9169-4628-9f61-baaef18a68a7" containerName="istio-proxy"
Nov 03 00:29:55.935782 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:55.935773    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba93722b-9169-4628-9f61-baaef18a68a7" containerName="istio-proxy"
Nov 03 00:29:55.940717 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:55.940701    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc"
Nov 03 00:29:55.959558 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:55.959531    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc"]
Nov 03 00:29:55.964090 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:55.964062    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-2df502\""
Nov 03 00:29:55.964090 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:55.964067    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-2df502-dockercfg-dcpcz\""
Nov 03 00:29:56.168958 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:56.159426    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sxbn\" (UniqueName: \"kubernetes.io/projected/1f594be4-6f2b-48f1-9fef-34a19645c6dc-kube-api-access-5sxbn\") pod \"payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc\" (UID: \"1f594be4-6f2b-48f1-9fef-34a19645c6dc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc"
Nov 03 00:29:56.168958 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:56.159513    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f594be4-6f2b-48f1-9fef-34a19645c6dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc\" (UID: \"1f594be4-6f2b-48f1-9fef-34a19645c6dc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc"
Nov 03 00:29:56.193941 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:56.193824    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl"]
Nov 03 00:29:56.197724 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:56.195302    5012 cpu_manager.go:414] "containerMap: removing container" podUID="ba93722b-9169-4628-9f61-baaef18a68a7" containerName="istio-proxy"
Nov 03 00:29:56.197724 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:56.195333    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba93722b-9169-4628-9f61-baaef18a68a7" containerName="istio-proxy"
Nov 03 00:29:56.197724 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:56.195618    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba93722b-9169-4628-9f61-baaef18a68a7" containerName="istio-proxy"
Nov 03 00:29:56.200191 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:56.200158    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl"
Nov 03 00:29:56.220209 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:56.219504    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl"]
Nov 03 00:29:56.261595 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:56.261062    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sxbn\" (UniqueName: \"kubernetes.io/projected/1f594be4-6f2b-48f1-9fef-34a19645c6dc-kube-api-access-5sxbn\") pod \"payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc\" (UID: \"1f594be4-6f2b-48f1-9fef-34a19645c6dc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc"
Nov 03 00:29:56.261595 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:56.261112    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83f374c7-f0aa-46c2-beca-544aaf65a7b7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl\" (UID: \"83f374c7-f0aa-46c2-beca-544aaf65a7b7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl"
Nov 03 00:29:56.261595 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:56.261228    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbm6g\" (UniqueName: \"kubernetes.io/projected/83f374c7-f0aa-46c2-beca-544aaf65a7b7-kube-api-access-gbm6g\") pod \"payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl\" (UID: \"83f374c7-f0aa-46c2-beca-544aaf65a7b7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl"
Nov 03 00:29:56.261595 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:56.261260    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f594be4-6f2b-48f1-9fef-34a19645c6dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc\" (UID: \"1f594be4-6f2b-48f1-9fef-34a19645c6dc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc"
Nov 03 00:29:56.262824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:56.262784    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f594be4-6f2b-48f1-9fef-34a19645c6dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc\" (UID: \"1f594be4-6f2b-48f1-9fef-34a19645c6dc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc"
Nov 03 00:29:56.280657 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:56.280614    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sxbn\" (UniqueName: \"kubernetes.io/projected/1f594be4-6f2b-48f1-9fef-34a19645c6dc-kube-api-access-5sxbn\") pod \"payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc\" (UID: \"1f594be4-6f2b-48f1-9fef-34a19645c6dc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc"
Nov 03 00:29:56.364491 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:56.364455    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83f374c7-f0aa-46c2-beca-544aaf65a7b7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl\" (UID: \"83f374c7-f0aa-46c2-beca-544aaf65a7b7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl"
Nov 03 00:29:56.364650 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:56.364541    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbm6g\" (UniqueName: \"kubernetes.io/projected/83f374c7-f0aa-46c2-beca-544aaf65a7b7-kube-api-access-gbm6g\") pod \"payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl\" (UID: \"83f374c7-f0aa-46c2-beca-544aaf65a7b7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl"
Nov 03 00:29:56.365363 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:56.365337    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83f374c7-f0aa-46c2-beca-544aaf65a7b7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl\" (UID: \"83f374c7-f0aa-46c2-beca-544aaf65a7b7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl"
Nov 03 00:29:56.382754 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:56.382729    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbm6g\" (UniqueName: \"kubernetes.io/projected/83f374c7-f0aa-46c2-beca-544aaf65a7b7-kube-api-access-gbm6g\") pod \"payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl\" (UID: \"83f374c7-f0aa-46c2-beca-544aaf65a7b7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl"
Nov 03 00:29:56.525703 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:56.525613    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl"
Nov 03 00:29:56.555210 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:56.555188    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc"
Nov 03 00:29:56.865889 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:29:56.865862    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f374c7_f0aa_46c2_beca_544aaf65a7b7.slice/crio-d77d25e8fa0b2e8e08e975eab127f938193cd41c35fe4db89a6730af07caf8b7 WatchSource:0}: Error finding container d77d25e8fa0b2e8e08e975eab127f938193cd41c35fe4db89a6730af07caf8b7: Status 404 returned error can't find the container with id d77d25e8fa0b2e8e08e975eab127f938193cd41c35fe4db89a6730af07caf8b7
Nov 03 00:29:56.870161 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:56.868854    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl"]
Nov 03 00:29:56.918107 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:29:56.918069    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f594be4_6f2b_48f1_9fef_34a19645c6dc.slice/crio-806b0cb1355c03752f5462a9043e10adcd047fc7ef054d1bb1d9a5f2e5a466f2 WatchSource:0}: Error finding container 806b0cb1355c03752f5462a9043e10adcd047fc7ef054d1bb1d9a5f2e5a466f2: Status 404 returned error can't find the container with id 806b0cb1355c03752f5462a9043e10adcd047fc7ef054d1bb1d9a5f2e5a466f2
Nov 03 00:29:56.918658 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:56.918602    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc"]
Nov 03 00:29:57.581500 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:57.581456    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc" event={"ID":"1f594be4-6f2b-48f1-9fef-34a19645c6dc","Type":"ContainerStarted","Data":"2cf89ab81baf4c5d2a55ce26bd7c05892309f068eb9c303bfbb0fae800d78710"}
Nov 03 00:29:57.582061 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:57.581531    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc" event={"ID":"1f594be4-6f2b-48f1-9fef-34a19645c6dc","Type":"ContainerStarted","Data":"806b0cb1355c03752f5462a9043e10adcd047fc7ef054d1bb1d9a5f2e5a466f2"}
Nov 03 00:29:57.582061 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:57.581564    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc"
Nov 03 00:29:57.592943 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:57.585714    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl" event={"ID":"83f374c7-f0aa-46c2-beca-544aaf65a7b7","Type":"ContainerStarted","Data":"d7b85ef8d446901729e625e36e030019bae1b989f2eed0dcd7cd744b9352f7fd"}
Nov 03 00:29:57.592943 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:57.585770    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl"
Nov 03 00:29:57.592943 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:57.585794    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl" event={"ID":"83f374c7-f0aa-46c2-beca-544aaf65a7b7","Type":"ContainerStarted","Data":"d77d25e8fa0b2e8e08e975eab127f938193cd41c35fe4db89a6730af07caf8b7"}
Nov 03 00:29:57.630492 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:57.630431    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc" podStartSLOduration=2.6304156 podStartE2EDuration="2.6304156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:57.626359891 +0000 UTC m=+1480.505943931" watchObservedRunningTime="2026-11-03 00:29:57.6304156 +0000 UTC m=+1480.509999695"
Nov 03 00:29:57.677847 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:57.677764    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl" podStartSLOduration=1.677751357 podStartE2EDuration="1.677751357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:57.674085885 +0000 UTC m=+1480.553669928" watchObservedRunningTime="2026-11-03 00:29:57.677751357 +0000 UTC m=+1480.557335399"
Nov 03 00:29:59.464150 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:59.464107    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff"
Nov 03 00:29:59.464947 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:59.464883    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t"
Nov 03 00:29:59.753144 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:59.753049    5012 patch_prober.go:29] interesting pod/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:29:59.753144 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:29:59.753125    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8" podUID="59529c4e-6121-4fcf-94c2-4671ab16d5e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:30:00.164233 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.164199    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j"]
Nov 03 00:30:00.170538 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.170510    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7"]
Nov 03 00:30:00.170658 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.170617    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j"
Nov 03 00:30:00.174684 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.174658    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7"
Nov 03 00:30:00.196169 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.194508    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4"]
Nov 03 00:30:00.202733 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.202506    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:30:00.204432 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.203980    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4"
Nov 03 00:30:00.212303 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.209046    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7"]
Nov 03 00:30:00.218101 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.215972    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j"]
Nov 03 00:30:00.225868 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.225846    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4"]
Nov 03 00:30:00.233432 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.233411    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp"]
Nov 03 00:30:00.233563 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.233542    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4"
Nov 03 00:30:00.240513 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.240467    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4"]
Nov 03 00:30:00.240513 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.240512    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4"]
Nov 03 00:30:00.240702 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.240579    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5125ef01-2234-4046-9b92-611898a84a28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7\" (UID: \"5125ef01-2234-4046-9b92-611898a84a28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7"
Nov 03 00:30:00.240702 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.240598    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp"
Nov 03 00:30:00.240702 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.240664    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksbhg\" (UniqueName: \"kubernetes.io/projected/5125ef01-2234-4046-9b92-611898a84a28-kube-api-access-ksbhg\") pod \"maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7\" (UID: \"5125ef01-2234-4046-9b92-611898a84a28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7"
Nov 03 00:30:00.240893 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.240831    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6822aa0-aaf5-4312-a392-f9441fc5b3ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4\" (UID: \"b6822aa0-aaf5-4312-a392-f9441fc5b3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4"
Nov 03 00:30:00.240985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.240941    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vbqb\" (UniqueName: \"kubernetes.io/projected/b91e9eb8-4deb-4cbe-bd7e-2e29d6849212-kube-api-access-9vbqb\") pod \"maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j\" (UID: \"b91e9eb8-4deb-4cbe-bd7e-2e29d6849212\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j"
Nov 03 00:30:00.241044 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.240983    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt46n\" (UniqueName: \"kubernetes.io/projected/b6822aa0-aaf5-4312-a392-f9441fc5b3ac-kube-api-access-nt46n\") pod \"maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4\" (UID: \"b6822aa0-aaf5-4312-a392-f9441fc5b3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4"
Nov 03 00:30:00.241098 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.241043    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91e9eb8-4deb-4cbe-bd7e-2e29d6849212-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j\" (UID: \"b91e9eb8-4deb-4cbe-bd7e-2e29d6849212\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j"
Nov 03 00:30:00.249745 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.249717    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp"]
Nov 03 00:30:00.342589 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.342555    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdxrw\" (UniqueName: \"kubernetes.io/projected/e716df36-50a9-414c-8ff4-73dca7eebd28-kube-api-access-cdxrw\") pod \"maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4\" (UID: \"e716df36-50a9-414c-8ff4-73dca7eebd28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4"
Nov 03 00:30:00.342735 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.342606    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5125ef01-2234-4046-9b92-611898a84a28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7\" (UID: \"5125ef01-2234-4046-9b92-611898a84a28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7"
Nov 03 00:30:00.342776 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.342746    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksbhg\" (UniqueName: \"kubernetes.io/projected/5125ef01-2234-4046-9b92-611898a84a28-kube-api-access-ksbhg\") pod \"maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7\" (UID: \"5125ef01-2234-4046-9b92-611898a84a28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7"
Nov 03 00:30:00.342819 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.342808    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkstt\" (UniqueName: \"kubernetes.io/projected/14d784a1-6d67-4a6d-9845-5e40e6e07c60-kube-api-access-jkstt\") pod \"maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp\" (UID: \"14d784a1-6d67-4a6d-9845-5e40e6e07c60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp"
Nov 03 00:30:00.342853 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.342846    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d784a1-6d67-4a6d-9845-5e40e6e07c60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp\" (UID: \"14d784a1-6d67-4a6d-9845-5e40e6e07c60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp"
Nov 03 00:30:00.342891 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.342872    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e716df36-50a9-414c-8ff4-73dca7eebd28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4\" (UID: \"e716df36-50a9-414c-8ff4-73dca7eebd28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4"
Nov 03 00:30:00.343015 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.342993    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6822aa0-aaf5-4312-a392-f9441fc5b3ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4\" (UID: \"b6822aa0-aaf5-4312-a392-f9441fc5b3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4"
Nov 03 00:30:00.343106 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.343085    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vbqb\" (UniqueName: \"kubernetes.io/projected/b91e9eb8-4deb-4cbe-bd7e-2e29d6849212-kube-api-access-9vbqb\") pod \"maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j\" (UID: \"b91e9eb8-4deb-4cbe-bd7e-2e29d6849212\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j"
Nov 03 00:30:00.343167 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.343132    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt46n\" (UniqueName: \"kubernetes.io/projected/b6822aa0-aaf5-4312-a392-f9441fc5b3ac-kube-api-access-nt46n\") pod \"maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4\" (UID: \"b6822aa0-aaf5-4312-a392-f9441fc5b3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4"
Nov 03 00:30:00.343209 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.343195    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91e9eb8-4deb-4cbe-bd7e-2e29d6849212-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j\" (UID: \"b91e9eb8-4deb-4cbe-bd7e-2e29d6849212\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j"
Nov 03 00:30:00.343391 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.343365    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5125ef01-2234-4046-9b92-611898a84a28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7\" (UID: \"5125ef01-2234-4046-9b92-611898a84a28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7"
Nov 03 00:30:00.343849 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.343813    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6822aa0-aaf5-4312-a392-f9441fc5b3ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4\" (UID: \"b6822aa0-aaf5-4312-a392-f9441fc5b3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4"
Nov 03 00:30:00.343996 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.343973    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91e9eb8-4deb-4cbe-bd7e-2e29d6849212-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j\" (UID: \"b91e9eb8-4deb-4cbe-bd7e-2e29d6849212\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j"
Nov 03 00:30:00.360839 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.360811    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt46n\" (UniqueName: \"kubernetes.io/projected/b6822aa0-aaf5-4312-a392-f9441fc5b3ac-kube-api-access-nt46n\") pod \"maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4\" (UID: \"b6822aa0-aaf5-4312-a392-f9441fc5b3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4"
Nov 03 00:30:00.360839 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.360825    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vbqb\" (UniqueName: \"kubernetes.io/projected/b91e9eb8-4deb-4cbe-bd7e-2e29d6849212-kube-api-access-9vbqb\") pod \"maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j\" (UID: \"b91e9eb8-4deb-4cbe-bd7e-2e29d6849212\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j"
Nov 03 00:30:00.360839 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.360831    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksbhg\" (UniqueName: \"kubernetes.io/projected/5125ef01-2234-4046-9b92-611898a84a28-kube-api-access-ksbhg\") pod \"maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7\" (UID: \"5125ef01-2234-4046-9b92-611898a84a28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7"
Nov 03 00:30:00.444786 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.444715    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdxrw\" (UniqueName: \"kubernetes.io/projected/e716df36-50a9-414c-8ff4-73dca7eebd28-kube-api-access-cdxrw\") pod \"maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4\" (UID: \"e716df36-50a9-414c-8ff4-73dca7eebd28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4"
Nov 03 00:30:00.444934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.444856    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkstt\" (UniqueName: \"kubernetes.io/projected/14d784a1-6d67-4a6d-9845-5e40e6e07c60-kube-api-access-jkstt\") pod \"maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp\" (UID: \"14d784a1-6d67-4a6d-9845-5e40e6e07c60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp"
Nov 03 00:30:00.444934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.444904    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d784a1-6d67-4a6d-9845-5e40e6e07c60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp\" (UID: \"14d784a1-6d67-4a6d-9845-5e40e6e07c60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp"
Nov 03 00:30:00.445055 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.444958    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e716df36-50a9-414c-8ff4-73dca7eebd28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4\" (UID: \"e716df36-50a9-414c-8ff4-73dca7eebd28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4"
Nov 03 00:30:00.445473 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.445452    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d784a1-6d67-4a6d-9845-5e40e6e07c60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp\" (UID: \"14d784a1-6d67-4a6d-9845-5e40e6e07c60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp"
Nov 03 00:30:00.445777 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.445754    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e716df36-50a9-414c-8ff4-73dca7eebd28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4\" (UID: \"e716df36-50a9-414c-8ff4-73dca7eebd28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4"
Nov 03 00:30:00.461877 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.461846    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdxrw\" (UniqueName: \"kubernetes.io/projected/e716df36-50a9-414c-8ff4-73dca7eebd28-kube-api-access-cdxrw\") pod \"maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4\" (UID: \"e716df36-50a9-414c-8ff4-73dca7eebd28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4"
Nov 03 00:30:00.462167 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.462144    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkstt\" (UniqueName: \"kubernetes.io/projected/14d784a1-6d67-4a6d-9845-5e40e6e07c60-kube-api-access-jkstt\") pod \"maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp\" (UID: \"14d784a1-6d67-4a6d-9845-5e40e6e07c60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp"
Nov 03 00:30:00.471202 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.471183    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm"
Nov 03 00:30:00.497733 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.497710    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j"
Nov 03 00:30:00.505509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.505484    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7"
Nov 03 00:30:00.529556 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.529520    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4"
Nov 03 00:30:00.568697 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.568646    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4"
Nov 03 00:30:00.573145 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.573091    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp"
Nov 03 00:30:00.836292 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:30:00.836253    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91e9eb8_4deb_4cbe_bd7e_2e29d6849212.slice/crio-8fe6884604f978f8ad34dbf607075cd5a77861d6b1d561c0a2a967df424f4d30 WatchSource:0}: Error finding container 8fe6884604f978f8ad34dbf607075cd5a77861d6b1d561c0a2a967df424f4d30: Status 404 returned error can't find the container with id 8fe6884604f978f8ad34dbf607075cd5a77861d6b1d561c0a2a967df424f4d30
Nov 03 00:30:00.846123 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.846098    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j"]
Nov 03 00:30:00.856148 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:30:00.856115    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5125ef01_2234_4046_9b92_611898a84a28.slice/crio-9bb98d000c51fa5de057b0481201edc3c9b85e40d21c85eceb282f57bb66814f WatchSource:0}: Error finding container 9bb98d000c51fa5de057b0481201edc3c9b85e40d21c85eceb282f57bb66814f: Status 404 returned error can't find the container with id 9bb98d000c51fa5de057b0481201edc3c9b85e40d21c85eceb282f57bb66814f
Nov 03 00:30:00.868445 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.868416    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7"]
Nov 03 00:30:00.887265 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:30:00.887237    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6822aa0_aaf5_4312_a392_f9441fc5b3ac.slice/crio-9326bcaf0604e588b5ab53175b55037443cca794c55ff6370b973a6dc7d10b24 WatchSource:0}: Error finding container 9326bcaf0604e588b5ab53175b55037443cca794c55ff6370b973a6dc7d10b24: Status 404 returned error can't find the container with id 9326bcaf0604e588b5ab53175b55037443cca794c55ff6370b973a6dc7d10b24
Nov 03 00:30:00.892999 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.892972    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4"]
Nov 03 00:30:00.924859 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:30:00.924836    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode716df36_50a9_414c_8ff4_73dca7eebd28.slice/crio-1ffa6ba50775b329d4f90b839e6278855c84990138d9dc5d1e022763285a3f6e WatchSource:0}: Error finding container 1ffa6ba50775b329d4f90b839e6278855c84990138d9dc5d1e022763285a3f6e: Status 404 returned error can't find the container with id 1ffa6ba50775b329d4f90b839e6278855c84990138d9dc5d1e022763285a3f6e
Nov 03 00:30:00.925302 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.925279    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4"]
Nov 03 00:30:00.942279 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:30:00.942251    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d784a1_6d67_4a6d_9845_5e40e6e07c60.slice/crio-c82c7a23f89432360a117943c7c3c8c29ce22d1abc6cf094568014f365521b03 WatchSource:0}: Error finding container c82c7a23f89432360a117943c7c3c8c29ce22d1abc6cf094568014f365521b03: Status 404 returned error can't find the container with id c82c7a23f89432360a117943c7c3c8c29ce22d1abc6cf094568014f365521b03
Nov 03 00:30:00.943828 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:00.943688    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp"]
Nov 03 00:30:01.634984 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:01.634946    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j" event={"ID":"b91e9eb8-4deb-4cbe-bd7e-2e29d6849212","Type":"ContainerStarted","Data":"59f95e9300e2407c7409c761fac9980746086ddf6e60994b3020668a6eeb69d6"}
Nov 03 00:30:01.635433 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:01.635003    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j" event={"ID":"b91e9eb8-4deb-4cbe-bd7e-2e29d6849212","Type":"ContainerStarted","Data":"8fe6884604f978f8ad34dbf607075cd5a77861d6b1d561c0a2a967df424f4d30"}
Nov 03 00:30:01.636777 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:01.636743    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7" event={"ID":"5125ef01-2234-4046-9b92-611898a84a28","Type":"ContainerStarted","Data":"23409c95a8a4cc995246554d2e366c1df082b83b6d2120d8e20c91b4f6334b78"}
Nov 03 00:30:01.636882 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:01.636803    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7" event={"ID":"5125ef01-2234-4046-9b92-611898a84a28","Type":"ContainerStarted","Data":"9bb98d000c51fa5de057b0481201edc3c9b85e40d21c85eceb282f57bb66814f"}
Nov 03 00:30:01.638589 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:01.638560    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp" event={"ID":"14d784a1-6d67-4a6d-9845-5e40e6e07c60","Type":"ContainerStarted","Data":"d9ff3ec8a10842a34c589f7a7329815bf5a7029db0ff6e29c1941c5462e65eec"}
Nov 03 00:30:01.638717 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:01.638613    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp" event={"ID":"14d784a1-6d67-4a6d-9845-5e40e6e07c60","Type":"ContainerStarted","Data":"c82c7a23f89432360a117943c7c3c8c29ce22d1abc6cf094568014f365521b03"}
Nov 03 00:30:01.640396 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:01.640369    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4" event={"ID":"e716df36-50a9-414c-8ff4-73dca7eebd28","Type":"ContainerStarted","Data":"60ce42d0a4bbc513c1eca3d4d3bd4a8e79adb05c46b85dcfed62157c076c8189"}
Nov 03 00:30:01.640489 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:01.640421    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4" event={"ID":"e716df36-50a9-414c-8ff4-73dca7eebd28","Type":"ContainerStarted","Data":"1ffa6ba50775b329d4f90b839e6278855c84990138d9dc5d1e022763285a3f6e"}
Nov 03 00:30:01.641977 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:01.641955    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4" event={"ID":"b6822aa0-aaf5-4312-a392-f9441fc5b3ac","Type":"ContainerStarted","Data":"c19a51a3fe4d87582787a1daaaad503d830aa6d47ac2d876239ba0452f243cdf"}
Nov 03 00:30:01.642050 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:01.641992    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4" event={"ID":"b6822aa0-aaf5-4312-a392-f9441fc5b3ac","Type":"ContainerStarted","Data":"9326bcaf0604e588b5ab53175b55037443cca794c55ff6370b973a6dc7d10b24"}
Nov 03 00:30:01.671746 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:01.671699    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j" podStartSLOduration=1.6716873589999999 podStartE2EDuration="1.671687359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:01.670629244 +0000 UTC m=+1484.550213314" watchObservedRunningTime="2026-11-03 00:30:01.671687359 +0000 UTC m=+1484.551271402"
Nov 03 00:30:01.705163 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:01.705111    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4" podStartSLOduration=1.705094428 podStartE2EDuration="1.705094428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:01.703456049 +0000 UTC m=+1484.583040092" watchObservedRunningTime="2026-11-03 00:30:01.705094428 +0000 UTC m=+1484.584678472"
Nov 03 00:30:01.742149 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:01.742099    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp" podStartSLOduration=1.7420858240000001 podStartE2EDuration="1.742085824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:01.739182255 +0000 UTC m=+1484.618766298" watchObservedRunningTime="2026-11-03 00:30:01.742085824 +0000 UTC m=+1484.621669863"
Nov 03 00:30:01.772876 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:01.772792    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7" podStartSLOduration=1.772779761 podStartE2EDuration="1.772779761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:01.771516568 +0000 UTC m=+1484.651100610" watchObservedRunningTime="2026-11-03 00:30:01.772779761 +0000 UTC m=+1484.652363805"
Nov 03 00:30:01.810428 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:01.809744    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4" podStartSLOduration=1.809731897 podStartE2EDuration="1.809731897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:01.805771364 +0000 UTC m=+1484.685355401" watchObservedRunningTime="2026-11-03 00:30:01.809731897 +0000 UTC m=+1484.689315938"
Nov 03 00:30:02.656162 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:02.656131    5012 generic.go:356] "Generic (PLEG): container finished" podID="b91e9eb8-4deb-4cbe-bd7e-2e29d6849212" containerID="59f95e9300e2407c7409c761fac9980746086ddf6e60994b3020668a6eeb69d6" exitCode=60
Nov 03 00:30:02.656630 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:02.656220    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j" event={"ID":"b91e9eb8-4deb-4cbe-bd7e-2e29d6849212","Type":"ContainerDied","Data":"59f95e9300e2407c7409c761fac9980746086ddf6e60994b3020668a6eeb69d6"}
Nov 03 00:30:02.656630 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:02.656421    5012 scope.go:122] "RemoveContainer" containerID="59f95e9300e2407c7409c761fac9980746086ddf6e60994b3020668a6eeb69d6"
Nov 03 00:30:02.657950 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:02.657923    5012 generic.go:356] "Generic (PLEG): container finished" podID="5125ef01-2234-4046-9b92-611898a84a28" containerID="23409c95a8a4cc995246554d2e366c1df082b83b6d2120d8e20c91b4f6334b78" exitCode=60
Nov 03 00:30:02.658167 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:02.657959    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7" event={"ID":"5125ef01-2234-4046-9b92-611898a84a28","Type":"ContainerDied","Data":"23409c95a8a4cc995246554d2e366c1df082b83b6d2120d8e20c91b4f6334b78"}
Nov 03 00:30:02.658239 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:02.658223    5012 scope.go:122] "RemoveContainer" containerID="23409c95a8a4cc995246554d2e366c1df082b83b6d2120d8e20c91b4f6334b78"
Nov 03 00:30:02.659722 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:02.659701    5012 generic.go:356] "Generic (PLEG): container finished" podID="14d784a1-6d67-4a6d-9845-5e40e6e07c60" containerID="d9ff3ec8a10842a34c589f7a7329815bf5a7029db0ff6e29c1941c5462e65eec" exitCode=60
Nov 03 00:30:02.660058 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:02.659945    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp" event={"ID":"14d784a1-6d67-4a6d-9845-5e40e6e07c60","Type":"ContainerDied","Data":"d9ff3ec8a10842a34c589f7a7329815bf5a7029db0ff6e29c1941c5462e65eec"}
Nov 03 00:30:02.660058 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:02.660011    5012 scope.go:122] "RemoveContainer" containerID="d9ff3ec8a10842a34c589f7a7329815bf5a7029db0ff6e29c1941c5462e65eec"
Nov 03 00:30:02.661703 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:02.661685    5012 generic.go:356] "Generic (PLEG): container finished" podID="59529c4e-6121-4fcf-94c2-4671ab16d5e2" containerID="1571366599c84892f1f34fef6db8205a271755c5d8e094464cf57108351e22d2" exitCode=0
Nov 03 00:30:02.661788 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:02.661751    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8" event={"ID":"59529c4e-6121-4fcf-94c2-4671ab16d5e2","Type":"ContainerDied","Data":"1571366599c84892f1f34fef6db8205a271755c5d8e094464cf57108351e22d2"}
Nov 03 00:30:02.662094 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:02.662076    5012 scope.go:122] "RemoveContainer" containerID="1571366599c84892f1f34fef6db8205a271755c5d8e094464cf57108351e22d2"
Nov 03 00:30:02.663358 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:02.663337    5012 generic.go:356] "Generic (PLEG): container finished" podID="e716df36-50a9-414c-8ff4-73dca7eebd28" containerID="60ce42d0a4bbc513c1eca3d4d3bd4a8e79adb05c46b85dcfed62157c076c8189" exitCode=60
Nov 03 00:30:02.663465 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:02.663416    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4" event={"ID":"e716df36-50a9-414c-8ff4-73dca7eebd28","Type":"ContainerDied","Data":"60ce42d0a4bbc513c1eca3d4d3bd4a8e79adb05c46b85dcfed62157c076c8189"}
Nov 03 00:30:02.663605 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:02.663591    5012 scope.go:122] "RemoveContainer" containerID="60ce42d0a4bbc513c1eca3d4d3bd4a8e79adb05c46b85dcfed62157c076c8189"
Nov 03 00:30:02.665082 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:02.665063    5012 generic.go:356] "Generic (PLEG): container finished" podID="b6822aa0-aaf5-4312-a392-f9441fc5b3ac" containerID="c19a51a3fe4d87582787a1daaaad503d830aa6d47ac2d876239ba0452f243cdf" exitCode=60
Nov 03 00:30:02.665180 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:02.665092    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4" event={"ID":"b6822aa0-aaf5-4312-a392-f9441fc5b3ac","Type":"ContainerDied","Data":"c19a51a3fe4d87582787a1daaaad503d830aa6d47ac2d876239ba0452f243cdf"}
Nov 03 00:30:02.665289 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:02.665277    5012 scope.go:122] "RemoveContainer" containerID="c19a51a3fe4d87582787a1daaaad503d830aa6d47ac2d876239ba0452f243cdf"
Nov 03 00:30:02.668560 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:02.668432    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:30:02.668560 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:02.668510    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:30:02.668560 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:02.668547    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:30:03.675822 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:03.675724    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4" event={"ID":"e716df36-50a9-414c-8ff4-73dca7eebd28","Type":"ContainerStarted","Data":"f0f4e30a467f900e72e8756786c7487c17d87f01d4f32591b622b2e0296d435f"}
Nov 03 00:30:03.677845 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:03.677819    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4" event={"ID":"b6822aa0-aaf5-4312-a392-f9441fc5b3ac","Type":"ContainerStarted","Data":"dae9eb2155a9a96163a3f5560e4505a0f4387dea95c5e4134077b95780ac0e3c"}
Nov 03 00:30:03.679692 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:03.679667    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j" event={"ID":"b91e9eb8-4deb-4cbe-bd7e-2e29d6849212","Type":"ContainerStarted","Data":"b30e4890d4f29537eb0e9e7f3aba9e108f071fdaf9e5cc0ec3f2663457c6f021"}
Nov 03 00:30:03.681556 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:03.681536    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7" event={"ID":"5125ef01-2234-4046-9b92-611898a84a28","Type":"ContainerStarted","Data":"de97b20340a4ecd051df5bba081fe7c729b6cb37c48177f6f6e669598c1ef758"}
Nov 03 00:30:03.683563 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:03.683538    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp" event={"ID":"14d784a1-6d67-4a6d-9845-5e40e6e07c60","Type":"ContainerStarted","Data":"c7b70ee439dcd8b60b43a992aa0fd9e74a6c6649facc2d09326f0c6c348a0198"}
Nov 03 00:30:03.685418 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:03.685399    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8" event={"ID":"59529c4e-6121-4fcf-94c2-4671ab16d5e2","Type":"ContainerStarted","Data":"ca5effc975fb64c83d56439a1d41011c0a670f442f520c66ba00967249c5bbe4"}
Nov 03 00:30:04.626934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:04.626185    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:30:04.626934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:04.626463    5012 patch_prober.go:29] interesting pod/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Nov 03 00:30:04.626934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:04.626519    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8" podUID="59529c4e-6121-4fcf-94c2-4671ab16d5e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Nov 03 00:30:04.708068 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:04.708029    5012 generic.go:356] "Generic (PLEG): container finished" podID="e716df36-50a9-414c-8ff4-73dca7eebd28" containerID="f0f4e30a467f900e72e8756786c7487c17d87f01d4f32591b622b2e0296d435f" exitCode=60
Nov 03 00:30:04.708662 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:04.708635    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4" event={"ID":"e716df36-50a9-414c-8ff4-73dca7eebd28","Type":"ContainerDied","Data":"f0f4e30a467f900e72e8756786c7487c17d87f01d4f32591b622b2e0296d435f"}
Nov 03 00:30:04.708831 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:04.708814    5012 scope.go:122] "RemoveContainer" containerID="60ce42d0a4bbc513c1eca3d4d3bd4a8e79adb05c46b85dcfed62157c076c8189"
Nov 03 00:30:04.709457 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:04.709438    5012 scope.go:122] "RemoveContainer" containerID="f0f4e30a467f900e72e8756786c7487c17d87f01d4f32591b622b2e0296d435f"
Nov 03 00:30:04.710345 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:30:04.710319    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4_odh-ai-gateway-infra(e716df36-50a9-414c-8ff4-73dca7eebd28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4" podUID="e716df36-50a9-414c-8ff4-73dca7eebd28"
Nov 03 00:30:04.757933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:04.746000    5012 generic.go:356] "Generic (PLEG): container finished" podID="b6822aa0-aaf5-4312-a392-f9441fc5b3ac" containerID="dae9eb2155a9a96163a3f5560e4505a0f4387dea95c5e4134077b95780ac0e3c" exitCode=60
Nov 03 00:30:04.757933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:04.746147    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4" event={"ID":"b6822aa0-aaf5-4312-a392-f9441fc5b3ac","Type":"ContainerDied","Data":"dae9eb2155a9a96163a3f5560e4505a0f4387dea95c5e4134077b95780ac0e3c"}
Nov 03 00:30:04.757933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:04.746666    5012 scope.go:122] "RemoveContainer" containerID="dae9eb2155a9a96163a3f5560e4505a0f4387dea95c5e4134077b95780ac0e3c"
Nov 03 00:30:04.757933 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:30:04.747055    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4_odh-ai-gateway-infra(b6822aa0-aaf5-4312-a392-f9441fc5b3ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4" podUID="b6822aa0-aaf5-4312-a392-f9441fc5b3ac"
Nov 03 00:30:04.786957 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:04.773680    5012 generic.go:356] "Generic (PLEG): container finished" podID="b91e9eb8-4deb-4cbe-bd7e-2e29d6849212" containerID="b30e4890d4f29537eb0e9e7f3aba9e108f071fdaf9e5cc0ec3f2663457c6f021" exitCode=60
Nov 03 00:30:04.786957 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:04.773810    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j" event={"ID":"b91e9eb8-4deb-4cbe-bd7e-2e29d6849212","Type":"ContainerDied","Data":"b30e4890d4f29537eb0e9e7f3aba9e108f071fdaf9e5cc0ec3f2663457c6f021"}
Nov 03 00:30:04.786957 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:04.774509    5012 scope.go:122] "RemoveContainer" containerID="b30e4890d4f29537eb0e9e7f3aba9e108f071fdaf9e5cc0ec3f2663457c6f021"
Nov 03 00:30:04.786957 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:30:04.775630    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j_odh-ai-gateway-infra(b91e9eb8-4deb-4cbe-bd7e-2e29d6849212)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j" podUID="b91e9eb8-4deb-4cbe-bd7e-2e29d6849212"
Nov 03 00:30:04.786957 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:04.778954    5012 generic.go:356] "Generic (PLEG): container finished" podID="5125ef01-2234-4046-9b92-611898a84a28" containerID="de97b20340a4ecd051df5bba081fe7c729b6cb37c48177f6f6e669598c1ef758" exitCode=60
Nov 03 00:30:04.786957 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:04.779105    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7" event={"ID":"5125ef01-2234-4046-9b92-611898a84a28","Type":"ContainerDied","Data":"de97b20340a4ecd051df5bba081fe7c729b6cb37c48177f6f6e669598c1ef758"}
Nov 03 00:30:04.786957 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:04.779528    5012 scope.go:122] "RemoveContainer" containerID="de97b20340a4ecd051df5bba081fe7c729b6cb37c48177f6f6e669598c1ef758"
Nov 03 00:30:04.786957 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:30:04.779886    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7_odh-ai-gateway-infra(5125ef01-2234-4046-9b92-611898a84a28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7" podUID="5125ef01-2234-4046-9b92-611898a84a28"
Nov 03 00:30:04.811932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:04.805373    5012 generic.go:356] "Generic (PLEG): container finished" podID="14d784a1-6d67-4a6d-9845-5e40e6e07c60" containerID="c7b70ee439dcd8b60b43a992aa0fd9e74a6c6649facc2d09326f0c6c348a0198" exitCode=60
Nov 03 00:30:04.811932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:04.805794    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp" event={"ID":"14d784a1-6d67-4a6d-9845-5e40e6e07c60","Type":"ContainerDied","Data":"c7b70ee439dcd8b60b43a992aa0fd9e74a6c6649facc2d09326f0c6c348a0198"}
Nov 03 00:30:04.811932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:04.810152    5012 scope.go:122] "RemoveContainer" containerID="c7b70ee439dcd8b60b43a992aa0fd9e74a6c6649facc2d09326f0c6c348a0198"
Nov 03 00:30:04.827956 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:30:04.820328    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp_odh-ai-gateway-infra(14d784a1-6d67-4a6d-9845-5e40e6e07c60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp" podUID="14d784a1-6d67-4a6d-9845-5e40e6e07c60"
Nov 03 00:30:04.903730 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:04.888961    5012 scope.go:122] "RemoveContainer" containerID="c19a51a3fe4d87582787a1daaaad503d830aa6d47ac2d876239ba0452f243cdf"
Nov 03 00:30:04.974527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:04.974407    5012 scope.go:122] "RemoveContainer" containerID="59f95e9300e2407c7409c761fac9980746086ddf6e60994b3020668a6eeb69d6"
Nov 03 00:30:05.015932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:05.015220    5012 scope.go:122] "RemoveContainer" containerID="23409c95a8a4cc995246554d2e366c1df082b83b6d2120d8e20c91b4f6334b78"
Nov 03 00:30:05.034603 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:05.034565    5012 scope.go:122] "RemoveContainer" containerID="d9ff3ec8a10842a34c589f7a7329815bf5a7029db0ff6e29c1941c5462e65eec"
Nov 03 00:30:05.623944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:05.623479    5012 patch_prober.go:29] interesting pod/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Nov 03 00:30:05.623944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:05.623540    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8" podUID="59529c4e-6121-4fcf-94c2-4671ab16d5e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Nov 03 00:30:05.821682 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:05.821655    5012 scope.go:122] "RemoveContainer" containerID="f0f4e30a467f900e72e8756786c7487c17d87f01d4f32591b622b2e0296d435f"
Nov 03 00:30:05.822158 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:30:05.822073    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4_odh-ai-gateway-infra(e716df36-50a9-414c-8ff4-73dca7eebd28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4" podUID="e716df36-50a9-414c-8ff4-73dca7eebd28"
Nov 03 00:30:05.823536 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:05.823512    5012 scope.go:122] "RemoveContainer" containerID="dae9eb2155a9a96163a3f5560e4505a0f4387dea95c5e4134077b95780ac0e3c"
Nov 03 00:30:05.823834 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:30:05.823806    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4_odh-ai-gateway-infra(b6822aa0-aaf5-4312-a392-f9441fc5b3ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4" podUID="b6822aa0-aaf5-4312-a392-f9441fc5b3ac"
Nov 03 00:30:05.825086 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:05.825067    5012 scope.go:122] "RemoveContainer" containerID="b30e4890d4f29537eb0e9e7f3aba9e108f071fdaf9e5cc0ec3f2663457c6f021"
Nov 03 00:30:05.825357 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:30:05.825340    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j_odh-ai-gateway-infra(b91e9eb8-4deb-4cbe-bd7e-2e29d6849212)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j" podUID="b91e9eb8-4deb-4cbe-bd7e-2e29d6849212"
Nov 03 00:30:05.826684 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:05.826666    5012 scope.go:122] "RemoveContainer" containerID="de97b20340a4ecd051df5bba081fe7c729b6cb37c48177f6f6e669598c1ef758"
Nov 03 00:30:05.826973 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:30:05.826952    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7_odh-ai-gateway-infra(5125ef01-2234-4046-9b92-611898a84a28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7" podUID="5125ef01-2234-4046-9b92-611898a84a28"
Nov 03 00:30:05.828257 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:05.828239    5012 scope.go:122] "RemoveContainer" containerID="c7b70ee439dcd8b60b43a992aa0fd9e74a6c6649facc2d09326f0c6c348a0198"
Nov 03 00:30:05.828503 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:30:05.828484    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp_odh-ai-gateway-infra(14d784a1-6d67-4a6d-9845-5e40e6e07c60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp" podUID="14d784a1-6d67-4a6d-9845-5e40e6e07c60"
Nov 03 00:30:06.620569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:06.620537    5012 patch_prober.go:29] interesting pod/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Nov 03 00:30:06.620785 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:06.620608    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8" podUID="59529c4e-6121-4fcf-94c2-4671ab16d5e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Nov 03 00:30:06.660390 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:06.660356    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx"]
Nov 03 00:30:06.664983 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:06.664962    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx"
Nov 03 00:30:06.678757 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:06.678734    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-ea275f-serving-cert\""
Nov 03 00:30:06.689998 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:06.689961    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx"]
Nov 03 00:30:06.819195 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:06.819139    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx\" (UID: \"a1d8aac2-8d32-4e6f-a342-d2e2c7425f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx"
Nov 03 00:30:06.819400 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:06.819351    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx\" (UID: \"a1d8aac2-8d32-4e6f-a342-d2e2c7425f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx"
Nov 03 00:30:06.819400 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:06.819396    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-metrics-certs\") pod \"maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx\" (UID: \"a1d8aac2-8d32-4e6f-a342-d2e2c7425f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx"
Nov 03 00:30:06.819598 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:06.819488    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdw55\" (UniqueName: \"kubernetes.io/projected/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-kube-api-access-cdw55\") pod \"maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx\" (UID: \"a1d8aac2-8d32-4e6f-a342-d2e2c7425f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx"
Nov 03 00:30:06.930003 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:06.923934    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx\" (UID: \"a1d8aac2-8d32-4e6f-a342-d2e2c7425f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx"
Nov 03 00:30:06.930003 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:06.924158    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx\" (UID: \"a1d8aac2-8d32-4e6f-a342-d2e2c7425f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx"
Nov 03 00:30:06.930003 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:06.924188    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-metrics-certs\") pod \"maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx\" (UID: \"a1d8aac2-8d32-4e6f-a342-d2e2c7425f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx"
Nov 03 00:30:06.930003 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:06.924263    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdw55\" (UniqueName: \"kubernetes.io/projected/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-kube-api-access-cdw55\") pod \"maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx\" (UID: \"a1d8aac2-8d32-4e6f-a342-d2e2c7425f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx"
Nov 03 00:30:06.930003 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:30:06.924712    5012 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-serving-cert: secret "maas-api-e2e-ipp-b-ea275f-serving-cert" not found
Nov 03 00:30:06.930003 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:30:06.924780    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-maas-api-tls podName:a1d8aac2-8d32-4e6f-a342-d2e2c7425f18 nodeName:}" failed. No retries permitted until 2026-11-03 00:30:07.424757731 +0000 UTC m=+1490.304341756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-maas-api-tls") pod "maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx" (UID: "a1d8aac2-8d32-4e6f-a342-d2e2c7425f18") : secret "maas-api-e2e-ipp-b-ea275f-serving-cert" not found
Nov 03 00:30:06.930003 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:06.925731    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx\" (UID: \"a1d8aac2-8d32-4e6f-a342-d2e2c7425f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx"
Nov 03 00:30:06.932046 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:06.931045    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-metrics-certs\") pod \"maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx\" (UID: \"a1d8aac2-8d32-4e6f-a342-d2e2c7425f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx"
Nov 03 00:30:06.946258 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:06.946224    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdw55\" (UniqueName: \"kubernetes.io/projected/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-kube-api-access-cdw55\") pod \"maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx\" (UID: \"a1d8aac2-8d32-4e6f-a342-d2e2c7425f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx"
Nov 03 00:30:07.364871 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:30:07.364838    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:30:07.430799 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:07.430758    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx\" (UID: \"a1d8aac2-8d32-4e6f-a342-d2e2c7425f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx"
Nov 03 00:30:07.433287 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:07.433259    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx\" (UID: \"a1d8aac2-8d32-4e6f-a342-d2e2c7425f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx"
Nov 03 00:30:07.579303 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:07.579271    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx"
Nov 03 00:30:07.624990 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:07.624027    5012 patch_prober.go:29] interesting pod/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Nov 03 00:30:07.624990 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:07.624090    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8" podUID="59529c4e-6121-4fcf-94c2-4671ab16d5e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Nov 03 00:30:07.965104 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:07.965068    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx"]
Nov 03 00:30:07.974659 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:30:07.974608    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d8aac2_8d32_4e6f_a342_d2e2c7425f18.slice/crio-317cce3a169db5cb65aa1e55fe3a1afcf7416189205917f2158370f0c9aa006d WatchSource:0}: Error finding container 317cce3a169db5cb65aa1e55fe3a1afcf7416189205917f2158370f0c9aa006d: Status 404 returned error can't find the container with id 317cce3a169db5cb65aa1e55fe3a1afcf7416189205917f2158370f0c9aa006d
Nov 03 00:30:08.614936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:08.604155    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc"
Nov 03 00:30:08.650313 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:08.645316    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl"
Nov 03 00:30:08.650313 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:08.647051    5012 patch_prober.go:29] interesting pod/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Nov 03 00:30:08.650313 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:08.647121    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8" podUID="59529c4e-6121-4fcf-94c2-4671ab16d5e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Nov 03 00:30:08.862336 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:08.862290    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx" event={"ID":"a1d8aac2-8d32-4e6f-a342-d2e2c7425f18","Type":"ContainerStarted","Data":"f03ee31151a91607cad779964acbaaadb4c22e193489b66051992944c4aa055f"}
Nov 03 00:30:08.862533 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:08.862360    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx" event={"ID":"a1d8aac2-8d32-4e6f-a342-d2e2c7425f18","Type":"ContainerStarted","Data":"317cce3a169db5cb65aa1e55fe3a1afcf7416189205917f2158370f0c9aa006d"}
Nov 03 00:30:08.862533 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:08.862403    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx"
Nov 03 00:30:09.625653 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:09.625619    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:30:09.626144 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:09.626123    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:30:09.626866 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:09.626840    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:30:09.677862 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:09.677818    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx" podStartSLOduration=3.244556143 podStartE2EDuration="3.677805389s" totalImagesPullingTime="433.249246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:08.913964951 +0000 UTC m=+1491.793548996" watchObservedRunningTime="2026-11-03 00:30:09.677805389 +0000 UTC m=+1492.557389432"
Nov 03 00:30:14.831297 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:14.831263    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v"]
Nov 03 00:30:14.837453 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:14.837423    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v"
Nov 03 00:30:14.867283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:14.867250    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-191362-serving-cert\""
Nov 03 00:30:14.867283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:14.867261    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v"]
Nov 03 00:30:14.881663 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:14.881622    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx"
Nov 03 00:30:14.973665 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:14.973635    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/907fc1f1-9a24-4941-a792-5d70a3bb3566-metrics-certs\") pod \"maas-api-e2e-praxis-b-191362-8579d8b649-59d6v\" (UID: \"907fc1f1-9a24-4941-a792-5d70a3bb3566\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v"
Nov 03 00:30:14.973832 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:14.973675    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/907fc1f1-9a24-4941-a792-5d70a3bb3566-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-191362-8579d8b649-59d6v\" (UID: \"907fc1f1-9a24-4941-a792-5d70a3bb3566\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v"
Nov 03 00:30:14.974107 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:14.974081    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg75k\" (UniqueName: \"kubernetes.io/projected/907fc1f1-9a24-4941-a792-5d70a3bb3566-kube-api-access-qg75k\") pod \"maas-api-e2e-praxis-b-191362-8579d8b649-59d6v\" (UID: \"907fc1f1-9a24-4941-a792-5d70a3bb3566\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v"
Nov 03 00:30:14.974213 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:14.974181    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/907fc1f1-9a24-4941-a792-5d70a3bb3566-maas-api-tls\") pod \"maas-api-e2e-praxis-b-191362-8579d8b649-59d6v\" (UID: \"907fc1f1-9a24-4941-a792-5d70a3bb3566\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v"
Nov 03 00:30:15.075385 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:15.075348    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg75k\" (UniqueName: \"kubernetes.io/projected/907fc1f1-9a24-4941-a792-5d70a3bb3566-kube-api-access-qg75k\") pod \"maas-api-e2e-praxis-b-191362-8579d8b649-59d6v\" (UID: \"907fc1f1-9a24-4941-a792-5d70a3bb3566\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v"
Nov 03 00:30:15.075542 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:15.075437    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/907fc1f1-9a24-4941-a792-5d70a3bb3566-maas-api-tls\") pod \"maas-api-e2e-praxis-b-191362-8579d8b649-59d6v\" (UID: \"907fc1f1-9a24-4941-a792-5d70a3bb3566\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v"
Nov 03 00:30:15.075542 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:15.075495    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/907fc1f1-9a24-4941-a792-5d70a3bb3566-metrics-certs\") pod \"maas-api-e2e-praxis-b-191362-8579d8b649-59d6v\" (UID: \"907fc1f1-9a24-4941-a792-5d70a3bb3566\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v"
Nov 03 00:30:15.075657 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:15.075556    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/907fc1f1-9a24-4941-a792-5d70a3bb3566-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-191362-8579d8b649-59d6v\" (UID: \"907fc1f1-9a24-4941-a792-5d70a3bb3566\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v"
Nov 03 00:30:15.076282 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:15.076250    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/907fc1f1-9a24-4941-a792-5d70a3bb3566-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-191362-8579d8b649-59d6v\" (UID: \"907fc1f1-9a24-4941-a792-5d70a3bb3566\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v"
Nov 03 00:30:15.077861 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:15.077833    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/907fc1f1-9a24-4941-a792-5d70a3bb3566-metrics-certs\") pod \"maas-api-e2e-praxis-b-191362-8579d8b649-59d6v\" (UID: \"907fc1f1-9a24-4941-a792-5d70a3bb3566\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v"
Nov 03 00:30:15.078025 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:15.077873    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/907fc1f1-9a24-4941-a792-5d70a3bb3566-maas-api-tls\") pod \"maas-api-e2e-praxis-b-191362-8579d8b649-59d6v\" (UID: \"907fc1f1-9a24-4941-a792-5d70a3bb3566\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v"
Nov 03 00:30:15.092888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:15.092836    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg75k\" (UniqueName: \"kubernetes.io/projected/907fc1f1-9a24-4941-a792-5d70a3bb3566-kube-api-access-qg75k\") pod \"maas-api-e2e-praxis-b-191362-8579d8b649-59d6v\" (UID: \"907fc1f1-9a24-4941-a792-5d70a3bb3566\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v"
Nov 03 00:30:15.175174 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:15.175140    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v"
Nov 03 00:30:15.487330 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:15.487296    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v"]
Nov 03 00:30:15.487481 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:30:15.487453    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907fc1f1_9a24_4941_a792_5d70a3bb3566.slice/crio-40fd600035a3c77b04aba9d326c7634c6eb364d0cb51de396f735e3ecb8d47df WatchSource:0}: Error finding container 40fd600035a3c77b04aba9d326c7634c6eb364d0cb51de396f735e3ecb8d47df: Status 404 returned error can't find the container with id 40fd600035a3c77b04aba9d326c7634c6eb364d0cb51de396f735e3ecb8d47df
Nov 03 00:30:15.931825 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:15.931790    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v" event={"ID":"907fc1f1-9a24-4941-a792-5d70a3bb3566","Type":"ContainerStarted","Data":"109647a616b4738d2f4051e01835ea8c4a1073ccd8f5ac33d00de9404d56b195"}
Nov 03 00:30:15.932207 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:15.931854    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v" event={"ID":"907fc1f1-9a24-4941-a792-5d70a3bb3566","Type":"ContainerStarted","Data":"40fd600035a3c77b04aba9d326c7634c6eb364d0cb51de396f735e3ecb8d47df"}
Nov 03 00:30:15.932207 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:15.931881    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v"
Nov 03 00:30:15.970992 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:15.970949    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v" podStartSLOduration=1.667098054 podStartE2EDuration="1.970935081s" totalImagesPullingTime="303.837027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:15.968969925 +0000 UTC m=+1498.848553964" watchObservedRunningTime="2026-11-03 00:30:15.970935081 +0000 UTC m=+1498.850519126"
Nov 03 00:30:17.243094 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:17.243063    5012 scope.go:122] "RemoveContainer" containerID="de97b20340a4ecd051df5bba081fe7c729b6cb37c48177f6f6e669598c1ef758"
Nov 03 00:30:17.243567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:17.243151    5012 scope.go:122] "RemoveContainer" containerID="b30e4890d4f29537eb0e9e7f3aba9e108f071fdaf9e5cc0ec3f2663457c6f021"
Nov 03 00:30:17.243567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:17.243249    5012 scope.go:122] "RemoveContainer" containerID="dae9eb2155a9a96163a3f5560e4505a0f4387dea95c5e4134077b95780ac0e3c"
Nov 03 00:30:17.800159 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:17.800129    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 00:30:17.805165 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:17.805144    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:30:17.806109 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:17.806088    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 00:30:17.810954 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:17.810939    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:30:17.966505 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:17.966469    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4" event={"ID":"b6822aa0-aaf5-4312-a392-f9441fc5b3ac","Type":"ContainerStarted","Data":"1bb749eb54b0b8c9289c5f59f0c583e318d38cde6cb70a7087786f66d6646dc1"}
Nov 03 00:30:17.968632 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:17.968605    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j" event={"ID":"b91e9eb8-4deb-4cbe-bd7e-2e29d6849212","Type":"ContainerStarted","Data":"f73a888bdf6700dccd4502a5de8fb1231b5d8ab4190662b5f8990328cf89f137"}
Nov 03 00:30:17.970612 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:17.970588    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7" event={"ID":"5125ef01-2234-4046-9b92-611898a84a28","Type":"ContainerStarted","Data":"68fe003d87acbc7e56f33b4c1448ebdbb7487e362f1731caf1443802473ab6e7"}
Nov 03 00:30:18.237532 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:18.237495    5012 scope.go:122] "RemoveContainer" containerID="c7b70ee439dcd8b60b43a992aa0fd9e74a6c6649facc2d09326f0c6c348a0198"
Nov 03 00:30:18.238497 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:18.238458    5012 scope.go:122] "RemoveContainer" containerID="f0f4e30a467f900e72e8756786c7487c17d87f01d4f32591b622b2e0296d435f"
Nov 03 00:30:19.013537 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:19.013490    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp" event={"ID":"14d784a1-6d67-4a6d-9845-5e40e6e07c60","Type":"ContainerStarted","Data":"3e27a78fb0ba386147c4f74968de21eba49f9bfa10cde9a5b6132c89e0257dee"}
Nov 03 00:30:19.016466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:19.016439    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4" event={"ID":"e716df36-50a9-414c-8ff4-73dca7eebd28","Type":"ContainerStarted","Data":"ab8a76412e69adcbe43ce73be75e59dd64884242d9dd4946f1f8591fd6c3c476"}
Nov 03 00:30:19.018740 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:19.018709    5012 generic.go:356] "Generic (PLEG): container finished" podID="b6822aa0-aaf5-4312-a392-f9441fc5b3ac" containerID="1bb749eb54b0b8c9289c5f59f0c583e318d38cde6cb70a7087786f66d6646dc1" exitCode=60
Nov 03 00:30:19.018864 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:19.018780    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4" event={"ID":"b6822aa0-aaf5-4312-a392-f9441fc5b3ac","Type":"ContainerDied","Data":"1bb749eb54b0b8c9289c5f59f0c583e318d38cde6cb70a7087786f66d6646dc1"}
Nov 03 00:30:19.018864 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:19.018840    5012 scope.go:122] "RemoveContainer" containerID="dae9eb2155a9a96163a3f5560e4505a0f4387dea95c5e4134077b95780ac0e3c"
Nov 03 00:30:19.019243 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:19.019224    5012 scope.go:122] "RemoveContainer" containerID="1bb749eb54b0b8c9289c5f59f0c583e318d38cde6cb70a7087786f66d6646dc1"
Nov 03 00:30:19.019617 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:30:19.019590    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4_odh-ai-gateway-infra(b6822aa0-aaf5-4312-a392-f9441fc5b3ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4" podUID="b6822aa0-aaf5-4312-a392-f9441fc5b3ac"
Nov 03 00:30:19.021537 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:19.021517    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4"]
Nov 03 00:30:19.022492 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:19.022472    5012 generic.go:356] "Generic (PLEG): container finished" podID="b91e9eb8-4deb-4cbe-bd7e-2e29d6849212" containerID="f73a888bdf6700dccd4502a5de8fb1231b5d8ab4190662b5f8990328cf89f137" exitCode=60
Nov 03 00:30:19.022602 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:19.022538    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j" event={"ID":"b91e9eb8-4deb-4cbe-bd7e-2e29d6849212","Type":"ContainerDied","Data":"f73a888bdf6700dccd4502a5de8fb1231b5d8ab4190662b5f8990328cf89f137"}
Nov 03 00:30:19.022901 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:19.022884    5012 scope.go:122] "RemoveContainer" containerID="f73a888bdf6700dccd4502a5de8fb1231b5d8ab4190662b5f8990328cf89f137"
Nov 03 00:30:19.023243 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:30:19.023223    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j_odh-ai-gateway-infra(b91e9eb8-4deb-4cbe-bd7e-2e29d6849212)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j" podUID="b91e9eb8-4deb-4cbe-bd7e-2e29d6849212"
Nov 03 00:30:19.024657 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:19.024634    5012 generic.go:356] "Generic (PLEG): container finished" podID="5125ef01-2234-4046-9b92-611898a84a28" containerID="68fe003d87acbc7e56f33b4c1448ebdbb7487e362f1731caf1443802473ab6e7" exitCode=60
Nov 03 00:30:19.024750 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:19.024688    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7" event={"ID":"5125ef01-2234-4046-9b92-611898a84a28","Type":"ContainerDied","Data":"68fe003d87acbc7e56f33b4c1448ebdbb7487e362f1731caf1443802473ab6e7"}
Nov 03 00:30:19.025024 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:19.025006    5012 scope.go:122] "RemoveContainer" containerID="68fe003d87acbc7e56f33b4c1448ebdbb7487e362f1731caf1443802473ab6e7"
Nov 03 00:30:19.025276 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:30:19.025259    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7_odh-ai-gateway-infra(5125ef01-2234-4046-9b92-611898a84a28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7" podUID="5125ef01-2234-4046-9b92-611898a84a28"
Nov 03 00:30:19.036180 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:19.036158    5012 scope.go:122] "RemoveContainer" containerID="b30e4890d4f29537eb0e9e7f3aba9e108f071fdaf9e5cc0ec3f2663457c6f021"
Nov 03 00:30:19.057746 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:19.057670    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7"]
Nov 03 00:30:19.060873 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:19.060856    5012 scope.go:122] "RemoveContainer" containerID="de97b20340a4ecd051df5bba081fe7c729b6cb37c48177f6f6e669598c1ef758"
Nov 03 00:30:19.094988 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:19.094929    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j"]
Nov 03 00:30:20.040066 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.040030    5012 generic.go:356] "Generic (PLEG): container finished" podID="14d784a1-6d67-4a6d-9845-5e40e6e07c60" containerID="3e27a78fb0ba386147c4f74968de21eba49f9bfa10cde9a5b6132c89e0257dee" exitCode=60
Nov 03 00:30:20.040505 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.040322    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp" event={"ID":"14d784a1-6d67-4a6d-9845-5e40e6e07c60","Type":"ContainerDied","Data":"3e27a78fb0ba386147c4f74968de21eba49f9bfa10cde9a5b6132c89e0257dee"}
Nov 03 00:30:20.040505 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.040384    5012 scope.go:122] "RemoveContainer" containerID="c7b70ee439dcd8b60b43a992aa0fd9e74a6c6649facc2d09326f0c6c348a0198"
Nov 03 00:30:20.040854 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.040818    5012 scope.go:122] "RemoveContainer" containerID="3e27a78fb0ba386147c4f74968de21eba49f9bfa10cde9a5b6132c89e0257dee"
Nov 03 00:30:20.041234 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:30:20.041208    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp_odh-ai-gateway-infra(14d784a1-6d67-4a6d-9845-5e40e6e07c60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp" podUID="14d784a1-6d67-4a6d-9845-5e40e6e07c60"
Nov 03 00:30:20.046300 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.046278    5012 generic.go:356] "Generic (PLEG): container finished" podID="e716df36-50a9-414c-8ff4-73dca7eebd28" containerID="ab8a76412e69adcbe43ce73be75e59dd64884242d9dd4946f1f8591fd6c3c476" exitCode=60
Nov 03 00:30:20.046401 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.046361    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4" event={"ID":"e716df36-50a9-414c-8ff4-73dca7eebd28","Type":"ContainerDied","Data":"ab8a76412e69adcbe43ce73be75e59dd64884242d9dd4946f1f8591fd6c3c476"}
Nov 03 00:30:20.046809 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.046777    5012 scope.go:122] "RemoveContainer" containerID="ab8a76412e69adcbe43ce73be75e59dd64884242d9dd4946f1f8591fd6c3c476"
Nov 03 00:30:20.047109 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:30:20.047082    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4_odh-ai-gateway-infra(e716df36-50a9-414c-8ff4-73dca7eebd28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4" podUID="e716df36-50a9-414c-8ff4-73dca7eebd28"
Nov 03 00:30:20.060312 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.060281    5012 scope.go:122] "RemoveContainer" containerID="f0f4e30a467f900e72e8756786c7487c17d87f01d4f32591b622b2e0296d435f"
Nov 03 00:30:20.076652 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.075586    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp"]
Nov 03 00:30:20.173953 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.173335    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4"]
Nov 03 00:30:20.280752 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.280719    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7"
Nov 03 00:30:20.376370 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.376341    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5125ef01-2234-4046-9b92-611898a84a28-kube-api-access-ksbhg\" (UniqueName: \"kubernetes.io/projected/5125ef01-2234-4046-9b92-611898a84a28-kube-api-access-ksbhg\") pod \"5125ef01-2234-4046-9b92-611898a84a28\" (UID: \"5125ef01-2234-4046-9b92-611898a84a28\") "
Nov 03 00:30:20.377539 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.377514    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5125ef01-2234-4046-9b92-611898a84a28-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5125ef01-2234-4046-9b92-611898a84a28-maas-api-service-ca\") pod \"5125ef01-2234-4046-9b92-611898a84a28\" (UID: \"5125ef01-2234-4046-9b92-611898a84a28\") "
Nov 03 00:30:20.378133 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.378077    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5125ef01-2234-4046-9b92-611898a84a28-maas-api-service-ca" pod "5125ef01-2234-4046-9b92-611898a84a28" (UID: "5125ef01-2234-4046-9b92-611898a84a28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:30:20.380978 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.380958    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5125ef01-2234-4046-9b92-611898a84a28-maas-api-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:20.384148 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.384127    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5125ef01-2234-4046-9b92-611898a84a28-kube-api-access-ksbhg" pod "5125ef01-2234-4046-9b92-611898a84a28" (UID: "5125ef01-2234-4046-9b92-611898a84a28"). InnerVolumeSpecName "kube-api-access-ksbhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:30:20.430066 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.430037    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4"
Nov 03 00:30:20.432343 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.432320    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j"
Nov 03 00:30:20.482581 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.482547    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6822aa0-aaf5-4312-a392-f9441fc5b3ac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6822aa0-aaf5-4312-a392-f9441fc5b3ac-maas-api-service-ca\") pod \"b6822aa0-aaf5-4312-a392-f9441fc5b3ac\" (UID: \"b6822aa0-aaf5-4312-a392-f9441fc5b3ac\") "
Nov 03 00:30:20.482776 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.482619    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6822aa0-aaf5-4312-a392-f9441fc5b3ac-kube-api-access-nt46n\" (UniqueName: \"kubernetes.io/projected/b6822aa0-aaf5-4312-a392-f9441fc5b3ac-kube-api-access-nt46n\") pod \"b6822aa0-aaf5-4312-a392-f9441fc5b3ac\" (UID: \"b6822aa0-aaf5-4312-a392-f9441fc5b3ac\") "
Nov 03 00:30:20.482776 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.482671    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b91e9eb8-4deb-4cbe-bd7e-2e29d6849212-kube-api-access-9vbqb\" (UniqueName: \"kubernetes.io/projected/b91e9eb8-4deb-4cbe-bd7e-2e29d6849212-kube-api-access-9vbqb\") pod \"b91e9eb8-4deb-4cbe-bd7e-2e29d6849212\" (UID: \"b91e9eb8-4deb-4cbe-bd7e-2e29d6849212\") "
Nov 03 00:30:20.482974 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.482949    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b91e9eb8-4deb-4cbe-bd7e-2e29d6849212-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91e9eb8-4deb-4cbe-bd7e-2e29d6849212-maas-api-service-ca\") pod \"b91e9eb8-4deb-4cbe-bd7e-2e29d6849212\" (UID: \"b91e9eb8-4deb-4cbe-bd7e-2e29d6849212\") "
Nov 03 00:30:20.483038 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.483007    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6822aa0-aaf5-4312-a392-f9441fc5b3ac-maas-api-service-ca" pod "b6822aa0-aaf5-4312-a392-f9441fc5b3ac" (UID: "b6822aa0-aaf5-4312-a392-f9441fc5b3ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:30:20.483416 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.483381    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b91e9eb8-4deb-4cbe-bd7e-2e29d6849212-maas-api-service-ca" pod "b91e9eb8-4deb-4cbe-bd7e-2e29d6849212" (UID: "b91e9eb8-4deb-4cbe-bd7e-2e29d6849212"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:30:20.483523 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.483477    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6822aa0-aaf5-4312-a392-f9441fc5b3ac-maas-api-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:20.483523 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.483494    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksbhg\" (UniqueName: \"kubernetes.io/projected/5125ef01-2234-4046-9b92-611898a84a28-kube-api-access-ksbhg\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:20.483523 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.483503    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91e9eb8-4deb-4cbe-bd7e-2e29d6849212-maas-api-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:20.485014 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.484986    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6822aa0-aaf5-4312-a392-f9441fc5b3ac-kube-api-access-nt46n" pod "b6822aa0-aaf5-4312-a392-f9441fc5b3ac" (UID: "b6822aa0-aaf5-4312-a392-f9441fc5b3ac"). InnerVolumeSpecName "kube-api-access-nt46n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:30:20.485424 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.485404    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b91e9eb8-4deb-4cbe-bd7e-2e29d6849212-kube-api-access-9vbqb" pod "b91e9eb8-4deb-4cbe-bd7e-2e29d6849212" (UID: "b91e9eb8-4deb-4cbe-bd7e-2e29d6849212"). InnerVolumeSpecName "kube-api-access-9vbqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:30:20.584770 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.584741    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt46n\" (UniqueName: \"kubernetes.io/projected/b6822aa0-aaf5-4312-a392-f9441fc5b3ac-kube-api-access-nt46n\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:20.584770 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:20.584770    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vbqb\" (UniqueName: \"kubernetes.io/projected/b91e9eb8-4deb-4cbe-bd7e-2e29d6849212-kube-api-access-9vbqb\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:21.062756 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.062727    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4"
Nov 03 00:30:21.063351 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.062763    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4" event={"ID":"b6822aa0-aaf5-4312-a392-f9441fc5b3ac","Type":"ContainerDied","Data":"9326bcaf0604e588b5ab53175b55037443cca794c55ff6370b973a6dc7d10b24"}
Nov 03 00:30:21.063351 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.062840    5012 scope.go:122] "RemoveContainer" containerID="1bb749eb54b0b8c9289c5f59f0c583e318d38cde6cb70a7087786f66d6646dc1"
Nov 03 00:30:21.066784 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.065897    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j" event={"ID":"b91e9eb8-4deb-4cbe-bd7e-2e29d6849212","Type":"ContainerDied","Data":"8fe6884604f978f8ad34dbf607075cd5a77861d6b1d561c0a2a967df424f4d30"}
Nov 03 00:30:21.066784 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.066000    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j"
Nov 03 00:30:21.068379 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.068106    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7"
Nov 03 00:30:21.068379 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.068133    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7" event={"ID":"5125ef01-2234-4046-9b92-611898a84a28","Type":"ContainerDied","Data":"9bb98d000c51fa5de057b0481201edc3c9b85e40d21c85eceb282f57bb66814f"}
Nov 03 00:30:21.124793 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.124766    5012 scope.go:122] "RemoveContainer" containerID="f73a888bdf6700dccd4502a5de8fb1231b5d8ab4190662b5f8990328cf89f137"
Nov 03 00:30:21.169165 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.169082    5012 scope.go:122] "RemoveContainer" containerID="68fe003d87acbc7e56f33b4c1448ebdbb7487e362f1731caf1443802473ab6e7"
Nov 03 00:30:21.174695 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.174665    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j"]
Nov 03 00:30:21.179957 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.179933    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894430-brn2j"]
Nov 03 00:30:21.248361 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.245391    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4"]
Nov 03 00:30:21.277609 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.277574    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b91e9eb8-4deb-4cbe-bd7e-2e29d6849212" path="/var/lib/kubelet/pods/b91e9eb8-4deb-4cbe-bd7e-2e29d6849212/volumes"
Nov 03 00:30:21.278099 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.278080    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4ae722-29894430-85fs4"]
Nov 03 00:30:21.330511 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.330484    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7"]
Nov 03 00:30:21.337846 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.337814    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a43474-29894430-4vwx7"]
Nov 03 00:30:21.371899 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.371871    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4"
Nov 03 00:30:21.424836 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.424464    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e716df36-50a9-414c-8ff4-73dca7eebd28-kube-api-access-cdxrw\" (UniqueName: \"kubernetes.io/projected/e716df36-50a9-414c-8ff4-73dca7eebd28-kube-api-access-cdxrw\") pod \"e716df36-50a9-414c-8ff4-73dca7eebd28\" (UID: \"e716df36-50a9-414c-8ff4-73dca7eebd28\") "
Nov 03 00:30:21.424836 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.424579    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e716df36-50a9-414c-8ff4-73dca7eebd28-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e716df36-50a9-414c-8ff4-73dca7eebd28-maas-api-service-ca\") pod \"e716df36-50a9-414c-8ff4-73dca7eebd28\" (UID: \"e716df36-50a9-414c-8ff4-73dca7eebd28\") "
Nov 03 00:30:21.425703 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.425671    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e716df36-50a9-414c-8ff4-73dca7eebd28-maas-api-service-ca" pod "e716df36-50a9-414c-8ff4-73dca7eebd28" (UID: "e716df36-50a9-414c-8ff4-73dca7eebd28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:30:21.433159 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.433128    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e716df36-50a9-414c-8ff4-73dca7eebd28-kube-api-access-cdxrw" pod "e716df36-50a9-414c-8ff4-73dca7eebd28" (UID: "e716df36-50a9-414c-8ff4-73dca7eebd28"). InnerVolumeSpecName "kube-api-access-cdxrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:30:21.462644 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.462615    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp"
Nov 03 00:30:21.528565 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.528517    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d784a1-6d67-4a6d-9845-5e40e6e07c60-kube-api-access-jkstt\" (UniqueName: \"kubernetes.io/projected/14d784a1-6d67-4a6d-9845-5e40e6e07c60-kube-api-access-jkstt\") pod \"14d784a1-6d67-4a6d-9845-5e40e6e07c60\" (UID: \"14d784a1-6d67-4a6d-9845-5e40e6e07c60\") "
Nov 03 00:30:21.528764 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.528581    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14d784a1-6d67-4a6d-9845-5e40e6e07c60-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d784a1-6d67-4a6d-9845-5e40e6e07c60-maas-api-service-ca\") pod \"14d784a1-6d67-4a6d-9845-5e40e6e07c60\" (UID: \"14d784a1-6d67-4a6d-9845-5e40e6e07c60\") "
Nov 03 00:30:21.549977 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.549945    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdxrw\" (UniqueName: \"kubernetes.io/projected/e716df36-50a9-414c-8ff4-73dca7eebd28-kube-api-access-cdxrw\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:21.550204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.550188    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e716df36-50a9-414c-8ff4-73dca7eebd28-maas-api-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:21.551689 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.551652    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d784a1-6d67-4a6d-9845-5e40e6e07c60-maas-api-service-ca" pod "14d784a1-6d67-4a6d-9845-5e40e6e07c60" (UID: "14d784a1-6d67-4a6d-9845-5e40e6e07c60"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:30:21.555630 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.555584    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d784a1-6d67-4a6d-9845-5e40e6e07c60-kube-api-access-jkstt" pod "14d784a1-6d67-4a6d-9845-5e40e6e07c60" (UID: "14d784a1-6d67-4a6d-9845-5e40e6e07c60"). InnerVolumeSpecName "kube-api-access-jkstt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:30:21.652652 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.652619    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkstt\" (UniqueName: \"kubernetes.io/projected/14d784a1-6d67-4a6d-9845-5e40e6e07c60-kube-api-access-jkstt\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:21.652652 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.652650    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d784a1-6d67-4a6d-9845-5e40e6e07c60-maas-api-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:21.962512 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:21.962426    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v"
Nov 03 00:30:22.111541 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:22.111511    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4"
Nov 03 00:30:22.112172 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:22.111505    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4" event={"ID":"e716df36-50a9-414c-8ff4-73dca7eebd28","Type":"ContainerDied","Data":"1ffa6ba50775b329d4f90b839e6278855c84990138d9dc5d1e022763285a3f6e"}
Nov 03 00:30:22.112328 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:22.112209    5012 scope.go:122] "RemoveContainer" containerID="ab8a76412e69adcbe43ce73be75e59dd64884242d9dd4946f1f8591fd6c3c476"
Nov 03 00:30:22.127305 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:22.127239    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp"
Nov 03 00:30:22.127305 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:22.127271    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp" event={"ID":"14d784a1-6d67-4a6d-9845-5e40e6e07c60","Type":"ContainerDied","Data":"c82c7a23f89432360a117943c7c3c8c29ce22d1abc6cf094568014f365521b03"}
Nov 03 00:30:22.133205 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:22.133184    5012 scope.go:122] "RemoveContainer" containerID="3e27a78fb0ba386147c4f74968de21eba49f9bfa10cde9a5b6132c89e0257dee"
Nov 03 00:30:22.178224 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:22.178200    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4"]
Nov 03 00:30:22.211309 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:22.211270    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-54f6d8-29894430-5npw4"]
Nov 03 00:30:22.236036 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:22.235965    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp"]
Nov 03 00:30:22.248516 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:22.248493    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-846492-29894430-kk2fp"]
Nov 03 00:30:23.245355 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:23.245325    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14d784a1-6d67-4a6d-9845-5e40e6e07c60" path="/var/lib/kubelet/pods/14d784a1-6d67-4a6d-9845-5e40e6e07c60/volumes"
Nov 03 00:30:23.245781 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:23.245756    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5125ef01-2234-4046-9b92-611898a84a28" path="/var/lib/kubelet/pods/5125ef01-2234-4046-9b92-611898a84a28/volumes"
Nov 03 00:30:23.246182 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:23.246169    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6822aa0-aaf5-4312-a392-f9441fc5b3ac" path="/var/lib/kubelet/pods/b6822aa0-aaf5-4312-a392-f9441fc5b3ac/volumes"
Nov 03 00:30:23.246867 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:23.246852    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e716df36-50a9-414c-8ff4-73dca7eebd28" path="/var/lib/kubelet/pods/e716df36-50a9-414c-8ff4-73dca7eebd28/volumes"
Nov 03 00:30:29.423105 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:29.423064    5012 patch_prober.go:29] interesting pod/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:30:29.423515 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:29.423136    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg" podUID="90b16caf-d83f-4c7b-a1fc-bdcb7d061653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:30:31.208943 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:31.208884    5012 generic.go:356] "Generic (PLEG): container finished" podID="90b16caf-d83f-4c7b-a1fc-bdcb7d061653" containerID="fd9b8edd5fd21bf21e78c6917f58e7c47fe4713f1a392f53830351f7ca1c6a7e" exitCode=0
Nov 03 00:30:31.209364 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:31.208949    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg" event={"ID":"90b16caf-d83f-4c7b-a1fc-bdcb7d061653","Type":"ContainerDied","Data":"fd9b8edd5fd21bf21e78c6917f58e7c47fe4713f1a392f53830351f7ca1c6a7e"}
Nov 03 00:30:31.209364 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:31.209332    5012 scope.go:122] "RemoveContainer" containerID="fd9b8edd5fd21bf21e78c6917f58e7c47fe4713f1a392f53830351f7ca1c6a7e"
Nov 03 00:30:31.210794 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:31.210769    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:30:31.210855 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:31.210835    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:30:31.210897 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:31.210885    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:30:32.220660 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:32.220622    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg" event={"ID":"90b16caf-d83f-4c7b-a1fc-bdcb7d061653","Type":"ContainerStarted","Data":"d49bd4b396cb8648aea74c1b4a1a1c8b5dc65af4f32f9e7c5aa22d3600ec3d75"}
Nov 03 00:30:32.732731 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:32.732698    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:30:32.732994 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:32.732895    5012 patch_prober.go:29] interesting pod/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 03 00:30:32.732994 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:32.732965    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg" podUID="90b16caf-d83f-4c7b-a1fc-bdcb7d061653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 03 00:30:33.732872 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:33.732837    5012 patch_prober.go:29] interesting pod/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 03 00:30:33.733320 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:33.732891    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg" podUID="90b16caf-d83f-4c7b-a1fc-bdcb7d061653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 03 00:30:34.733109 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:34.733073    5012 patch_prober.go:29] interesting pod/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 03 00:30:34.733527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:34.733138    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg" podUID="90b16caf-d83f-4c7b-a1fc-bdcb7d061653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 03 00:30:35.015546 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:35.015460    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"]
Nov 03 00:30:35.015826 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:35.015678    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf" podUID="19edcff3-65d5-489c-bb33-7ad5e4bcc791" containerName="istio-proxy" containerID="cri-o://d1f9a1cce5798b8c619e50aca38ae8e03916658b119fbb673716f828abc6ff06" gracePeriod=30
Nov 03 00:30:35.023432 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:35.023401    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"]
Nov 03 00:30:35.023815 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:35.023787    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m" podUID="188d122e-1edb-44d5-8c9c-0119c8bb4e3b" containerName="istio-proxy" containerID="cri-o://3c7905aa0cdfca4f747d30b5e5ef20170c69f5ed732a52bf797e07ed1c41bd46" gracePeriod=30
Nov 03 00:30:35.733431 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:35.733383    5012 patch_prober.go:29] interesting pod/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 03 00:30:35.733813 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:35.733441    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg" podUID="90b16caf-d83f-4c7b-a1fc-bdcb7d061653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 03 00:30:36.733520 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:36.733488    5012 patch_prober.go:29] interesting pod/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 03 00:30:36.733927 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:36.733543    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg" podUID="90b16caf-d83f-4c7b-a1fc-bdcb7d061653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 03 00:30:37.379888 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:30:37.379817    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:30:37.738667 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:37.738573    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:30:37.739420 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:37.739395    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:30:37.741662 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:37.741640    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:30:40.119373 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.119336    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t"]
Nov 03 00:30:40.354308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.354268    5012 generic.go:356] "Generic (PLEG): container finished" podID="19edcff3-65d5-489c-bb33-7ad5e4bcc791" containerID="d1f9a1cce5798b8c619e50aca38ae8e03916658b119fbb673716f828abc6ff06" exitCode=0
Nov 03 00:30:40.354604 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.354576    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf" event={"ID":"19edcff3-65d5-489c-bb33-7ad5e4bcc791","Type":"ContainerDied","Data":"d1f9a1cce5798b8c619e50aca38ae8e03916658b119fbb673716f828abc6ff06"}
Nov 03 00:30:40.358422 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.358395    5012 generic.go:356] "Generic (PLEG): container finished" podID="188d122e-1edb-44d5-8c9c-0119c8bb4e3b" containerID="3c7905aa0cdfca4f747d30b5e5ef20170c69f5ed732a52bf797e07ed1c41bd46" exitCode=0
Nov 03 00:30:40.358632 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.358611    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m" event={"ID":"188d122e-1edb-44d5-8c9c-0119c8bb4e3b","Type":"ContainerDied","Data":"3c7905aa0cdfca4f747d30b5e5ef20170c69f5ed732a52bf797e07ed1c41bd46"}
Nov 03 00:30:40.425548 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.425254    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:30:40.428868 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.428831    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:30:40.471015 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.470988    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-podinfo\") pod \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") "
Nov 03 00:30:40.471193 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.471031    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-workload-socket\") pod \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") "
Nov 03 00:30:40.471193 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.471124    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-credential-socket\") pod \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") "
Nov 03 00:30:40.471193 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.471167    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-token\" (UniqueName: \"kubernetes.io/projected/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-token\") pod \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") "
Nov 03 00:30:40.471383 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.471195    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-envoy\") pod \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") "
Nov 03 00:30:40.471383 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.471294    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-workload-certs\") pod \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") "
Nov 03 00:30:40.471383 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.471345    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19edcff3-65d5-489c-bb33-7ad5e4bcc791-kube-api-access-m57hn\" (UniqueName: \"kubernetes.io/projected/19edcff3-65d5-489c-bb33-7ad5e4bcc791-kube-api-access-m57hn\") pod \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") "
Nov 03 00:30:40.471550 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.471382    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-data\") pod \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") "
Nov 03 00:30:40.471550 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.471385    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-workload-socket" pod "188d122e-1edb-44d5-8c9c-0119c8bb4e3b" (UID: "188d122e-1edb-44d5-8c9c-0119c8bb4e3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:30:40.471550 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.471449    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-podinfo\") pod \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") "
Nov 03 00:30:40.471550 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.471502    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istiod-ca-cert\") pod \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") "
Nov 03 00:30:40.471550 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.471511    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-credential-socket" pod "19edcff3-65d5-489c-bb33-7ad5e4bcc791" (UID: "19edcff3-65d5-489c-bb33-7ad5e4bcc791"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:30:40.471550 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.471544    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-workload-certs\") pod \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") "
Nov 03 00:30:40.471892 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.471585    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-data\") pod \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") "
Nov 03 00:30:40.471892 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.471623    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-credential-socket\") pod \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") "
Nov 03 00:30:40.471892 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.471677    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-envoy\") pod \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") "
Nov 03 00:30:40.471892 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.471687    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-workload-certs" pod "188d122e-1edb-44d5-8c9c-0119c8bb4e3b" (UID: "188d122e-1edb-44d5-8c9c-0119c8bb4e3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:30:40.471892 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.471751    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-token\" (UniqueName: \"kubernetes.io/projected/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-token\") pod \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") "
Nov 03 00:30:40.471892 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.471801    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-data" pod "188d122e-1edb-44d5-8c9c-0119c8bb4e3b" (UID: "188d122e-1edb-44d5-8c9c-0119c8bb4e3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:30:40.471892 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.471809    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-kube-api-access-2fn6h\" (UniqueName: \"kubernetes.io/projected/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-kube-api-access-2fn6h\") pod \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") "
Nov 03 00:30:40.471892 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.471872    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-workload-socket\") pod \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\" (UID: \"19edcff3-65d5-489c-bb33-7ad5e4bcc791\") "
Nov 03 00:30:40.472332 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.471957    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istiod-ca-cert\") pod \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\" (UID: \"188d122e-1edb-44d5-8c9c-0119c8bb4e3b\") "
Nov 03 00:30:40.472332 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.472004    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-workload-certs" pod "19edcff3-65d5-489c-bb33-7ad5e4bcc791" (UID: "19edcff3-65d5-489c-bb33-7ad5e4bcc791"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:30:40.472449 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.472330    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istiod-ca-cert" pod "19edcff3-65d5-489c-bb33-7ad5e4bcc791" (UID: "19edcff3-65d5-489c-bb33-7ad5e4bcc791"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:30:40.472555 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.472534    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:40.472619 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.472555    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:40.472619 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.472564    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:40.472619 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.472572    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:40.472619 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.472580    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:40.472619 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.472588    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:40.472619 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.472559    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-workload-socket" pod "19edcff3-65d5-489c-bb33-7ad5e4bcc791" (UID: "19edcff3-65d5-489c-bb33-7ad5e4bcc791"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:30:40.473022 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.472741    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-credential-socket" pod "188d122e-1edb-44d5-8c9c-0119c8bb4e3b" (UID: "188d122e-1edb-44d5-8c9c-0119c8bb4e3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:30:40.473022 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.472942    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-data" pod "19edcff3-65d5-489c-bb33-7ad5e4bcc791" (UID: "19edcff3-65d5-489c-bb33-7ad5e4bcc791"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:30:40.473196 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.473065    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istiod-ca-cert" pod "188d122e-1edb-44d5-8c9c-0119c8bb4e3b" (UID: "188d122e-1edb-44d5-8c9c-0119c8bb4e3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:30:40.473757 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.473736    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-envoy" pod "188d122e-1edb-44d5-8c9c-0119c8bb4e3b" (UID: "188d122e-1edb-44d5-8c9c-0119c8bb4e3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:30:40.473757 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.473732    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-podinfo" pod "19edcff3-65d5-489c-bb33-7ad5e4bcc791" (UID: "19edcff3-65d5-489c-bb33-7ad5e4bcc791"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:30:40.475161 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.475133    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-envoy" pod "19edcff3-65d5-489c-bb33-7ad5e4bcc791" (UID: "19edcff3-65d5-489c-bb33-7ad5e4bcc791"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:30:40.475281 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.475160    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19edcff3-65d5-489c-bb33-7ad5e4bcc791-kube-api-access-m57hn" pod "19edcff3-65d5-489c-bb33-7ad5e4bcc791" (UID: "19edcff3-65d5-489c-bb33-7ad5e4bcc791"). InnerVolumeSpecName "kube-api-access-m57hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:30:40.475657 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.475632    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-podinfo" pod "188d122e-1edb-44d5-8c9c-0119c8bb4e3b" (UID: "188d122e-1edb-44d5-8c9c-0119c8bb4e3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:30:40.475760 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.475698    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-token" pod "188d122e-1edb-44d5-8c9c-0119c8bb4e3b" (UID: "188d122e-1edb-44d5-8c9c-0119c8bb4e3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:30:40.476682 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.476657    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-token" pod "19edcff3-65d5-489c-bb33-7ad5e4bcc791" (UID: "19edcff3-65d5-489c-bb33-7ad5e4bcc791"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:30:40.477020 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.476997    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-kube-api-access-2fn6h" pod "188d122e-1edb-44d5-8c9c-0119c8bb4e3b" (UID: "188d122e-1edb-44d5-8c9c-0119c8bb4e3b"). InnerVolumeSpecName "kube-api-access-2fn6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:30:40.573622 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.573587    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:40.573622 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.573614    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:40.573622 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.573625    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:40.573840 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.573634    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:40.573840 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.573644    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:40.573840 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.573652    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fn6h\" (UniqueName: \"kubernetes.io/projected/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-kube-api-access-2fn6h\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:40.573840 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.573660    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19edcff3-65d5-489c-bb33-7ad5e4bcc791-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:40.573840 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.573670    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:40.573840 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.573679    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19edcff3-65d5-489c-bb33-7ad5e4bcc791-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:40.573840 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.573687    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:40.573840 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.573696    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/188d122e-1edb-44d5-8c9c-0119c8bb4e3b-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:40.573840 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:40.573704    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m57hn\" (UniqueName: \"kubernetes.io/projected/19edcff3-65d5-489c-bb33-7ad5e4bcc791-kube-api-access-m57hn\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:41.370248 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:41.370218    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"
Nov 03 00:30:41.370660 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:41.370251    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m" event={"ID":"188d122e-1edb-44d5-8c9c-0119c8bb4e3b","Type":"ContainerDied","Data":"be5f1c41594022b60ba7e8a4e2cdd7e8221f1944dfa24c50e9ae76d96a0d02d2"}
Nov 03 00:30:41.370660 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:41.370319    5012 scope.go:122] "RemoveContainer" containerID="3c7905aa0cdfca4f747d30b5e5ef20170c69f5ed732a52bf797e07ed1c41bd46"
Nov 03 00:30:41.372068 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:41.372044    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf" event={"ID":"19edcff3-65d5-489c-bb33-7ad5e4bcc791","Type":"ContainerDied","Data":"f4e2be576e01737fad6045b38af12a0f61c27c0e0f3be6cdd87d89f608af5275"}
Nov 03 00:30:41.372173 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:41.372079    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"
Nov 03 00:30:41.388398 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:41.388370    5012 scope.go:122] "RemoveContainer" containerID="d1f9a1cce5798b8c619e50aca38ae8e03916658b119fbb673716f828abc6ff06"
Nov 03 00:30:41.432472 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:41.432438    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"]
Nov 03 00:30:41.448478 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:41.448441    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-7fw2m"]
Nov 03 00:30:41.468800 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:41.468772    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"]
Nov 03 00:30:41.475587 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:41.475556    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rgwqf"]
Nov 03 00:30:43.245828 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:43.245797    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="188d122e-1edb-44d5-8c9c-0119c8bb4e3b" path="/var/lib/kubelet/pods/188d122e-1edb-44d5-8c9c-0119c8bb4e3b/volumes"
Nov 03 00:30:43.246328 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:43.246312    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19edcff3-65d5-489c-bb33-7ad5e4bcc791" path="/var/lib/kubelet/pods/19edcff3-65d5-489c-bb33-7ad5e4bcc791/volumes"
Nov 03 00:30:44.923470 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.923434    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q"]
Nov 03 00:30:44.924608 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924576    5012 cpu_manager.go:414] "containerMap: removing container" podUID="e716df36-50a9-414c-8ff4-73dca7eebd28" containerName="cleanup"
Nov 03 00:30:44.924608 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924603    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e716df36-50a9-414c-8ff4-73dca7eebd28" containerName="cleanup"
Nov 03 00:30:44.924798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924620    5012 cpu_manager.go:414] "containerMap: removing container" podUID="188d122e-1edb-44d5-8c9c-0119c8bb4e3b" containerName="istio-proxy"
Nov 03 00:30:44.924798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924628    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="188d122e-1edb-44d5-8c9c-0119c8bb4e3b" containerName="istio-proxy"
Nov 03 00:30:44.924798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924650    5012 cpu_manager.go:414] "containerMap: removing container" podUID="5125ef01-2234-4046-9b92-611898a84a28" containerName="cleanup"
Nov 03 00:30:44.924798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924657    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5125ef01-2234-4046-9b92-611898a84a28" containerName="cleanup"
Nov 03 00:30:44.924798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924672    5012 cpu_manager.go:414] "containerMap: removing container" podUID="e716df36-50a9-414c-8ff4-73dca7eebd28" containerName="cleanup"
Nov 03 00:30:44.924798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924679    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e716df36-50a9-414c-8ff4-73dca7eebd28" containerName="cleanup"
Nov 03 00:30:44.924798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924693    5012 cpu_manager.go:414] "containerMap: removing container" podUID="14d784a1-6d67-4a6d-9845-5e40e6e07c60" containerName="cleanup"
Nov 03 00:30:44.924798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924701    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14d784a1-6d67-4a6d-9845-5e40e6e07c60" containerName="cleanup"
Nov 03 00:30:44.924798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924716    5012 cpu_manager.go:414] "containerMap: removing container" podUID="e716df36-50a9-414c-8ff4-73dca7eebd28" containerName="cleanup"
Nov 03 00:30:44.924798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924723    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e716df36-50a9-414c-8ff4-73dca7eebd28" containerName="cleanup"
Nov 03 00:30:44.924798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924737    5012 cpu_manager.go:414] "containerMap: removing container" podUID="b91e9eb8-4deb-4cbe-bd7e-2e29d6849212" containerName="cleanup"
Nov 03 00:30:44.924798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924744    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b91e9eb8-4deb-4cbe-bd7e-2e29d6849212" containerName="cleanup"
Nov 03 00:30:44.924798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924767    5012 cpu_manager.go:414] "containerMap: removing container" podUID="b6822aa0-aaf5-4312-a392-f9441fc5b3ac" containerName="cleanup"
Nov 03 00:30:44.924798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924775    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6822aa0-aaf5-4312-a392-f9441fc5b3ac" containerName="cleanup"
Nov 03 00:30:44.924798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924791    5012 cpu_manager.go:414] "containerMap: removing container" podUID="b6822aa0-aaf5-4312-a392-f9441fc5b3ac" containerName="cleanup"
Nov 03 00:30:44.924798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924799    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6822aa0-aaf5-4312-a392-f9441fc5b3ac" containerName="cleanup"
Nov 03 00:30:44.925942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924812    5012 cpu_manager.go:414] "containerMap: removing container" podUID="b6822aa0-aaf5-4312-a392-f9441fc5b3ac" containerName="cleanup"
Nov 03 00:30:44.925942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924818    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6822aa0-aaf5-4312-a392-f9441fc5b3ac" containerName="cleanup"
Nov 03 00:30:44.925942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924832    5012 cpu_manager.go:414] "containerMap: removing container" podUID="19edcff3-65d5-489c-bb33-7ad5e4bcc791" containerName="istio-proxy"
Nov 03 00:30:44.925942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924840    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19edcff3-65d5-489c-bb33-7ad5e4bcc791" containerName="istio-proxy"
Nov 03 00:30:44.925942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924853    5012 cpu_manager.go:414] "containerMap: removing container" podUID="5125ef01-2234-4046-9b92-611898a84a28" containerName="cleanup"
Nov 03 00:30:44.925942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924860    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5125ef01-2234-4046-9b92-611898a84a28" containerName="cleanup"
Nov 03 00:30:44.925942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924881    5012 cpu_manager.go:414] "containerMap: removing container" podUID="14d784a1-6d67-4a6d-9845-5e40e6e07c60" containerName="cleanup"
Nov 03 00:30:44.925942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924887    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14d784a1-6d67-4a6d-9845-5e40e6e07c60" containerName="cleanup"
Nov 03 00:30:44.925942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924926    5012 cpu_manager.go:414] "containerMap: removing container" podUID="b91e9eb8-4deb-4cbe-bd7e-2e29d6849212" containerName="cleanup"
Nov 03 00:30:44.925942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.924933    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b91e9eb8-4deb-4cbe-bd7e-2e29d6849212" containerName="cleanup"
Nov 03 00:30:44.925942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.925132    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="5125ef01-2234-4046-9b92-611898a84a28" containerName="cleanup"
Nov 03 00:30:44.925942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.925146    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="e716df36-50a9-414c-8ff4-73dca7eebd28" containerName="cleanup"
Nov 03 00:30:44.925942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.925156    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="14d784a1-6d67-4a6d-9845-5e40e6e07c60" containerName="cleanup"
Nov 03 00:30:44.925942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.925167    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="e716df36-50a9-414c-8ff4-73dca7eebd28" containerName="cleanup"
Nov 03 00:30:44.925942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.925179    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="b91e9eb8-4deb-4cbe-bd7e-2e29d6849212" containerName="cleanup"
Nov 03 00:30:44.925942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.925191    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6822aa0-aaf5-4312-a392-f9441fc5b3ac" containerName="cleanup"
Nov 03 00:30:44.925942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.925199    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6822aa0-aaf5-4312-a392-f9441fc5b3ac" containerName="cleanup"
Nov 03 00:30:44.925942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.925208    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6822aa0-aaf5-4312-a392-f9441fc5b3ac" containerName="cleanup"
Nov 03 00:30:44.925942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.925217    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="19edcff3-65d5-489c-bb33-7ad5e4bcc791" containerName="istio-proxy"
Nov 03 00:30:44.925942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.925228    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="5125ef01-2234-4046-9b92-611898a84a28" containerName="cleanup"
Nov 03 00:30:44.925942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.925237    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="14d784a1-6d67-4a6d-9845-5e40e6e07c60" containerName="cleanup"
Nov 03 00:30:44.925942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.925248    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="b91e9eb8-4deb-4cbe-bd7e-2e29d6849212" containerName="cleanup"
Nov 03 00:30:44.925942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.925408    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="188d122e-1edb-44d5-8c9c-0119c8bb4e3b" containerName="istio-proxy"
Nov 03 00:30:44.934072 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.934049    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q"
Nov 03 00:30:44.941785 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.941741    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q"]
Nov 03 00:30:44.952407 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.949548    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2af7b0\""
Nov 03 00:30:44.969315 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.969286    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqbb5\" (UniqueName: \"kubernetes.io/projected/ef6038ec-0566-4dd7-adb6-33ac116c4c75-kube-api-access-bqbb5\") pod \"payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q\" (UID: \"ef6038ec-0566-4dd7-adb6-33ac116c4c75\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q"
Nov 03 00:30:44.969604 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.969576    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6038ec-0566-4dd7-adb6-33ac116c4c75-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q\" (UID: \"ef6038ec-0566-4dd7-adb6-33ac116c4c75\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q"
Nov 03 00:30:44.971207 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:44.971184    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2af7b0-dockercfg-pz97x\""
Nov 03 00:30:45.071134 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.071101    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqbb5\" (UniqueName: \"kubernetes.io/projected/ef6038ec-0566-4dd7-adb6-33ac116c4c75-kube-api-access-bqbb5\") pod \"payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q\" (UID: \"ef6038ec-0566-4dd7-adb6-33ac116c4c75\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q"
Nov 03 00:30:45.071299 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.071200    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6038ec-0566-4dd7-adb6-33ac116c4c75-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q\" (UID: \"ef6038ec-0566-4dd7-adb6-33ac116c4c75\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q"
Nov 03 00:30:45.072110 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.072087    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6038ec-0566-4dd7-adb6-33ac116c4c75-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q\" (UID: \"ef6038ec-0566-4dd7-adb6-33ac116c4c75\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q"
Nov 03 00:30:45.115527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.115495    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqbb5\" (UniqueName: \"kubernetes.io/projected/ef6038ec-0566-4dd7-adb6-33ac116c4c75-kube-api-access-bqbb5\") pod \"payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q\" (UID: \"ef6038ec-0566-4dd7-adb6-33ac116c4c75\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q"
Nov 03 00:30:45.120096 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.120054    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t" podUID="09c04970-f7a0-401e-a077-79a5012fb572" containerName="payload-pre-processing" containerID="cri-o://6057f25729ea95d92f9080a9d1c3924ee893f853e2bcd4d8cf502e76570e12df" gracePeriod=25
Nov 03 00:30:45.263288 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.263256    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q"
Nov 03 00:30:45.419175 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.419138    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr"]
Nov 03 00:30:45.420021 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.419998    5012 cpu_manager.go:414] "containerMap: removing container" podUID="14d784a1-6d67-4a6d-9845-5e40e6e07c60" containerName="cleanup"
Nov 03 00:30:45.420164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.420020    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14d784a1-6d67-4a6d-9845-5e40e6e07c60" containerName="cleanup"
Nov 03 00:30:45.420164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.420041    5012 cpu_manager.go:414] "containerMap: removing container" podUID="b91e9eb8-4deb-4cbe-bd7e-2e29d6849212" containerName="cleanup"
Nov 03 00:30:45.420164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.420049    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b91e9eb8-4deb-4cbe-bd7e-2e29d6849212" containerName="cleanup"
Nov 03 00:30:45.420164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.420083    5012 cpu_manager.go:414] "containerMap: removing container" podUID="5125ef01-2234-4046-9b92-611898a84a28" containerName="cleanup"
Nov 03 00:30:45.420164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.420090    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5125ef01-2234-4046-9b92-611898a84a28" containerName="cleanup"
Nov 03 00:30:45.420426 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.420271    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="14d784a1-6d67-4a6d-9845-5e40e6e07c60" containerName="cleanup"
Nov 03 00:30:45.420426 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.420287    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="e716df36-50a9-414c-8ff4-73dca7eebd28" containerName="cleanup"
Nov 03 00:30:45.420426 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.420300    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="b91e9eb8-4deb-4cbe-bd7e-2e29d6849212" containerName="cleanup"
Nov 03 00:30:45.420426 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.420315    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="5125ef01-2234-4046-9b92-611898a84a28" containerName="cleanup"
Nov 03 00:30:45.434102 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.434075    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr"
Nov 03 00:30:45.471721 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.461662    5012 generic.go:356] "Generic (PLEG): container finished" podID="09c04970-f7a0-401e-a077-79a5012fb572" containerID="6057f25729ea95d92f9080a9d1c3924ee893f853e2bcd4d8cf502e76570e12df" exitCode=0
Nov 03 00:30:45.471721 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.461733    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t" event={"ID":"09c04970-f7a0-401e-a077-79a5012fb572","Type":"ContainerDied","Data":"6057f25729ea95d92f9080a9d1c3924ee893f853e2bcd4d8cf502e76570e12df"}
Nov 03 00:30:45.471721 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.461781    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t" event={"ID":"09c04970-f7a0-401e-a077-79a5012fb572","Type":"ContainerDied","Data":"09e55772a81a21ca539a121a0605c2a3d93275256927a2e4c2503177066438ae"}
Nov 03 00:30:45.471721 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.461802    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09e55772a81a21ca539a121a0605c2a3d93275256927a2e4c2503177066438ae"
Nov 03 00:30:45.471721 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.461818    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr"]
Nov 03 00:30:45.476424 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.476084    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t"
Nov 03 00:30:45.482183 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.479521    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09c04970-f7a0-401e-a077-79a5012fb572-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09c04970-f7a0-401e-a077-79a5012fb572-plugins-config-volume\") pod \"09c04970-f7a0-401e-a077-79a5012fb572\" (UID: \"09c04970-f7a0-401e-a077-79a5012fb572\") "
Nov 03 00:30:45.482183 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.479725    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c04970-f7a0-401e-a077-79a5012fb572-kube-api-access-6kjtq\" (UniqueName: \"kubernetes.io/projected/09c04970-f7a0-401e-a077-79a5012fb572-kube-api-access-6kjtq\") pod \"09c04970-f7a0-401e-a077-79a5012fb572\" (UID: \"09c04970-f7a0-401e-a077-79a5012fb572\") "
Nov 03 00:30:45.482183 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.480147    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj97k\" (UniqueName: \"kubernetes.io/projected/fb399f46-4a86-4660-bfef-c6267eaff189-kube-api-access-vj97k\") pod \"payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr\" (UID: \"fb399f46-4a86-4660-bfef-c6267eaff189\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr"
Nov 03 00:30:45.482183 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.480218    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb399f46-4a86-4660-bfef-c6267eaff189-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr\" (UID: \"fb399f46-4a86-4660-bfef-c6267eaff189\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr"
Nov 03 00:30:45.482183 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.480621    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c04970-f7a0-401e-a077-79a5012fb572-plugins-config-volume" pod "09c04970-f7a0-401e-a077-79a5012fb572" (UID: "09c04970-f7a0-401e-a077-79a5012fb572"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:30:45.500969 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.486112    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c04970-f7a0-401e-a077-79a5012fb572-kube-api-access-6kjtq" pod "09c04970-f7a0-401e-a077-79a5012fb572" (UID: "09c04970-f7a0-401e-a077-79a5012fb572"). InnerVolumeSpecName "kube-api-access-6kjtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:30:45.581876 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.581837    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj97k\" (UniqueName: \"kubernetes.io/projected/fb399f46-4a86-4660-bfef-c6267eaff189-kube-api-access-vj97k\") pod \"payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr\" (UID: \"fb399f46-4a86-4660-bfef-c6267eaff189\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr"
Nov 03 00:30:45.582083 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.581983    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb399f46-4a86-4660-bfef-c6267eaff189-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr\" (UID: \"fb399f46-4a86-4660-bfef-c6267eaff189\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr"
Nov 03 00:30:45.582194 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.582172    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kjtq\" (UniqueName: \"kubernetes.io/projected/09c04970-f7a0-401e-a077-79a5012fb572-kube-api-access-6kjtq\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:45.582268 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.582203    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09c04970-f7a0-401e-a077-79a5012fb572-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:30:45.584235 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.582880    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb399f46-4a86-4660-bfef-c6267eaff189-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr\" (UID: \"fb399f46-4a86-4660-bfef-c6267eaff189\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr"
Nov 03 00:30:45.616752 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.616229    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj97k\" (UniqueName: \"kubernetes.io/projected/fb399f46-4a86-4660-bfef-c6267eaff189-kube-api-access-vj97k\") pod \"payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr\" (UID: \"fb399f46-4a86-4660-bfef-c6267eaff189\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr"
Nov 03 00:30:45.617961 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:30:45.617932    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6038ec_0566_4dd7_adb6_33ac116c4c75.slice/crio-0cecd3f820e6e633eca69df5f19758b8dd39fa2c6363a9fd1bed9bc0184c5c48 WatchSource:0}: Error finding container 0cecd3f820e6e633eca69df5f19758b8dd39fa2c6363a9fd1bed9bc0184c5c48: Status 404 returned error can't find the container with id 0cecd3f820e6e633eca69df5f19758b8dd39fa2c6363a9fd1bed9bc0184c5c48
Nov 03 00:30:45.619187 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.619158    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q"]
Nov 03 00:30:45.814023 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:45.813894    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr"
Nov 03 00:30:46.120802 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:46.120772    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr"]
Nov 03 00:30:46.122495 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:30:46.122462    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb399f46_4a86_4660_bfef_c6267eaff189.slice/crio-77324d81d44619c139610a7480f5be4b741743bf075786ac9ddab432a83b8a2f WatchSource:0}: Error finding container 77324d81d44619c139610a7480f5be4b741743bf075786ac9ddab432a83b8a2f: Status 404 returned error can't find the container with id 77324d81d44619c139610a7480f5be4b741743bf075786ac9ddab432a83b8a2f
Nov 03 00:30:46.474336 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:46.474244    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr" event={"ID":"fb399f46-4a86-4660-bfef-c6267eaff189","Type":"ContainerStarted","Data":"3d49eb56f198e697eae77dace597f0cc8c30f9716fbdcc1bae30472126814f0a"}
Nov 03 00:30:46.474336 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:46.474306    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr" event={"ID":"fb399f46-4a86-4660-bfef-c6267eaff189","Type":"ContainerStarted","Data":"77324d81d44619c139610a7480f5be4b741743bf075786ac9ddab432a83b8a2f"}
Nov 03 00:30:46.474336 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:46.474336    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr"
Nov 03 00:30:46.476084 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:46.476051    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q" event={"ID":"ef6038ec-0566-4dd7-adb6-33ac116c4c75","Type":"ContainerStarted","Data":"4edb58b558d867f1001a71202ebaf481ff9d5b65896025b72e8fb35b512ec144"}
Nov 03 00:30:46.476243 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:46.476108    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q" event={"ID":"ef6038ec-0566-4dd7-adb6-33ac116c4c75","Type":"ContainerStarted","Data":"0cecd3f820e6e633eca69df5f19758b8dd39fa2c6363a9fd1bed9bc0184c5c48"}
Nov 03 00:30:46.476243 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:46.476120    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t"
Nov 03 00:30:46.476243 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:46.476136    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q"
Nov 03 00:30:46.560090 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:46.560038    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr" podStartSLOduration=1.560022778 podStartE2EDuration="1.560022778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:46.520868921 +0000 UTC m=+1529.400452970" watchObservedRunningTime="2026-11-03 00:30:46.560022778 +0000 UTC m=+1529.439606821"
Nov 03 00:30:46.561670 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:46.561641    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t"]
Nov 03 00:30:46.565671 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:46.565649    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-77ff69dc5-q2c7t"]
Nov 03 00:30:46.620898 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:46.620837    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q" podStartSLOduration=2.620820631 podStartE2EDuration="2.620820631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:46.615111379 +0000 UTC m=+1529.494695423" watchObservedRunningTime="2026-11-03 00:30:46.620820631 +0000 UTC m=+1529.500404679"
Nov 03 00:30:47.296932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:47.289966    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09c04970-f7a0-401e-a077-79a5012fb572" path="/var/lib/kubelet/pods/09c04970-f7a0-401e-a077-79a5012fb572/volumes"
Nov 03 00:30:51.510062 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:51.510016    5012 patch_prober.go:29] interesting pod/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:30:51.510452 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:51.510084    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b" podUID="5138f419-8f8a-4402-b134-8fdc698d5de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:30:52.530899 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:52.530866    5012 generic.go:356] "Generic (PLEG): container finished" podID="5138f419-8f8a-4402-b134-8fdc698d5de5" containerID="2e35845ac19b31c9a771449c8d3bacefb3048e0579961b3e09315714544715cc" exitCode=0
Nov 03 00:30:52.531369 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:52.530953    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b" event={"ID":"5138f419-8f8a-4402-b134-8fdc698d5de5","Type":"ContainerDied","Data":"2e35845ac19b31c9a771449c8d3bacefb3048e0579961b3e09315714544715cc"}
Nov 03 00:30:52.531369 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:52.531327    5012 scope.go:122] "RemoveContainer" containerID="2e35845ac19b31c9a771449c8d3bacefb3048e0579961b3e09315714544715cc"
Nov 03 00:30:52.532718 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:52.532693    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:30:52.532783 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:52.532759    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:30:52.532820 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:52.532792    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:30:53.541421 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:53.541384    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b" event={"ID":"5138f419-8f8a-4402-b134-8fdc698d5de5","Type":"ContainerStarted","Data":"0b3e7ca758f193346e814a881ec8468b42a11c95ad52039c0de120c7234e89a5"}
Nov 03 00:30:53.857047 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:53.857014    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:30:53.857217 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:53.857195    5012 patch_prober.go:29] interesting pod/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Nov 03 00:30:53.857288 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:53.857245    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b" podUID="5138f419-8f8a-4402-b134-8fdc698d5de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Nov 03 00:30:54.856748 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:54.856711    5012 patch_prober.go:29] interesting pod/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Nov 03 00:30:54.857178 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:54.856775    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b" podUID="5138f419-8f8a-4402-b134-8fdc698d5de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Nov 03 00:30:55.857273 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:55.857234    5012 patch_prober.go:29] interesting pod/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Nov 03 00:30:55.857759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:55.857304    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b" podUID="5138f419-8f8a-4402-b134-8fdc698d5de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Nov 03 00:30:56.857146 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:56.857103    5012 patch_prober.go:29] interesting pod/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Nov 03 00:30:56.857350 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:56.857175    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b" podUID="5138f419-8f8a-4402-b134-8fdc698d5de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Nov 03 00:30:57.512523 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:57.511353    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q"
Nov 03 00:30:57.512523 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:57.512089    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr"
Nov 03 00:30:57.857115 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:57.857082    5012 patch_prober.go:29] interesting pod/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Nov 03 00:30:57.857283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:57.857132    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b" podUID="5138f419-8f8a-4402-b134-8fdc698d5de5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Nov 03 00:30:58.222260 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.221408    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"]
Nov 03 00:30:58.222741 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.222708    5012 cpu_manager.go:414] "containerMap: removing container" podUID="09c04970-f7a0-401e-a077-79a5012fb572" containerName="payload-pre-processing"
Nov 03 00:30:58.222741 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.222731    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c04970-f7a0-401e-a077-79a5012fb572" containerName="payload-pre-processing"
Nov 03 00:30:58.222940 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.222922    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c04970-f7a0-401e-a077-79a5012fb572" containerName="payload-pre-processing"
Nov 03 00:30:58.227762 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.227743    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:30:58.228540 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.228512    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"]
Nov 03 00:30:58.242861 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.241454    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-4ae722\"/\"openshift-service-ca.crt\""
Nov 03 00:30:58.242861 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.241820    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-4ae722\"/\"kube-root-ca.crt\""
Nov 03 00:30:58.254593 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.251940    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-4ae722\"/\"default-dockercfg-fzzbm\""
Nov 03 00:30:58.254593 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.251977    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-4ae722\"/\"praxis-route-4ae722-kserve-self-signed-certs\""
Nov 03 00:30:58.352276 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.352247    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-tls-certs\") pod \"praxis-route-4ae722-kserve-84d945f5f-clg9n\" (UID: \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\") " pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:30:58.352434 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.352380    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-home\") pod \"praxis-route-4ae722-kserve-84d945f5f-clg9n\" (UID: \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\") " pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:30:58.352508 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.352488    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r26w\" (UniqueName: \"kubernetes.io/projected/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-kube-api-access-9r26w\") pod \"praxis-route-4ae722-kserve-84d945f5f-clg9n\" (UID: \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\") " pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:30:58.352556 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.352527    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-model-cache\") pod \"praxis-route-4ae722-kserve-84d945f5f-clg9n\" (UID: \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\") " pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:30:58.352556 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.352552    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-tmp-dir\") pod \"praxis-route-4ae722-kserve-84d945f5f-clg9n\" (UID: \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\") " pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:30:58.352643 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.352584    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-dshm\") pod \"praxis-route-4ae722-kserve-84d945f5f-clg9n\" (UID: \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\") " pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:30:58.466151 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.466114    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-tls-certs\") pod \"praxis-route-4ae722-kserve-84d945f5f-clg9n\" (UID: \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\") " pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:30:58.466321 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.466238    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-home\") pod \"praxis-route-4ae722-kserve-84d945f5f-clg9n\" (UID: \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\") " pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:30:58.466376 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.466345    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r26w\" (UniqueName: \"kubernetes.io/projected/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-kube-api-access-9r26w\") pod \"praxis-route-4ae722-kserve-84d945f5f-clg9n\" (UID: \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\") " pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:30:58.466424 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.466386    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-model-cache\") pod \"praxis-route-4ae722-kserve-84d945f5f-clg9n\" (UID: \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\") " pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:30:58.466490 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.466430    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-tmp-dir\") pod \"praxis-route-4ae722-kserve-84d945f5f-clg9n\" (UID: \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\") " pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:30:58.466550 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.466500    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-dshm\") pod \"praxis-route-4ae722-kserve-84d945f5f-clg9n\" (UID: \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\") " pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:30:58.466846 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.466822    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-home\") pod \"praxis-route-4ae722-kserve-84d945f5f-clg9n\" (UID: \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\") " pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:30:58.467124 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.467105    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-model-cache\") pod \"praxis-route-4ae722-kserve-84d945f5f-clg9n\" (UID: \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\") " pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:30:58.467206 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.466822    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-tmp-dir\") pod \"praxis-route-4ae722-kserve-84d945f5f-clg9n\" (UID: \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\") " pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:30:58.471961 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.471853    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-tls-certs\") pod \"praxis-route-4ae722-kserve-84d945f5f-clg9n\" (UID: \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\") " pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:30:58.472237 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.472212    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-dshm\") pod \"praxis-route-4ae722-kserve-84d945f5f-clg9n\" (UID: \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\") " pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:30:58.633247 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.607961    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r26w\" (UniqueName: \"kubernetes.io/projected/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-kube-api-access-9r26w\") pod \"praxis-route-4ae722-kserve-84d945f5f-clg9n\" (UID: \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\") " pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:30:58.844663 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:58.844632    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:30:59.024127 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:59.023760    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:30:59.024317 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:59.024291    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:30:59.027559 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:59.027536    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:30:59.163024 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:59.162991    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"]
Nov 03 00:30:59.163417 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:30:59.163389    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02a9089a_7dee_4192_b1c3_3ccc297c8d4f.slice/crio-42a7c45ace38f841dde842f67cfac952354fbae7cacae928531e9078b8caff1a WatchSource:0}: Error finding container 42a7c45ace38f841dde842f67cfac952354fbae7cacae928531e9078b8caff1a: Status 404 returned error can't find the container with id 42a7c45ace38f841dde842f67cfac952354fbae7cacae928531e9078b8caff1a
Nov 03 00:30:59.654682 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:59.654646    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n" event={"ID":"02a9089a-7dee-4192-b1c3-3ccc297c8d4f","Type":"ContainerStarted","Data":"b7dff37044ce76116760ee47170d9a6dff6d02291fe8cd690a5129aa36f40a3a"}
Nov 03 00:30:59.655202 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:59.654696    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n" event={"ID":"02a9089a-7dee-4192-b1c3-3ccc297c8d4f","Type":"ContainerStarted","Data":"42a7c45ace38f841dde842f67cfac952354fbae7cacae928531e9078b8caff1a"}
Nov 03 00:30:59.691832 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:59.691788    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n" podStartSLOduration=1.6917750150000002 podStartE2EDuration="1.691775015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:59.68944353 +0000 UTC m=+1542.569027574" watchObservedRunningTime="2026-11-03 00:30:59.691775015 +0000 UTC m=+1542.571359057"
Nov 03 00:30:59.845264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:30:59.845231    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:31:02.953483 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:02.953453    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-885tl_df410080-3e08-4f8d-b1d3-84db851d8782/payload-processing/0.log"
Nov 03 00:31:03.174229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:03.174200    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-885tl_df410080-3e08-4f8d-b1d3-84db851d8782/payload-processing/0.log"
Nov 03 00:31:03.362627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:03.362597    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr_fb399f46-4a86-4660-bfef-c6267eaff189/payload-processing/0.log"
Nov 03 00:31:07.369156 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:31:07.369116    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:31:07.516771 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:07.516734    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl"]
Nov 03 00:31:07.529640 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:07.526308    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl"
Nov 03 00:31:07.548849 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:07.548814    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ea275f\""
Nov 03 00:31:07.549120 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:07.548943    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ea275f-dockercfg-dbqrk\""
Nov 03 00:31:07.580498 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:07.580464    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl"]
Nov 03 00:31:07.609422 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:07.609394    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7404d00f-242b-4f97-a9c6-bb04c79d7348-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl\" (UID: \"7404d00f-242b-4f97-a9c6-bb04c79d7348\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl"
Nov 03 00:31:07.609601 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:07.609573    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngzr4\" (UniqueName: \"kubernetes.io/projected/7404d00f-242b-4f97-a9c6-bb04c79d7348-kube-api-access-ngzr4\") pod \"payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl\" (UID: \"7404d00f-242b-4f97-a9c6-bb04c79d7348\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl"
Nov 03 00:31:07.711282 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:07.711200    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngzr4\" (UniqueName: \"kubernetes.io/projected/7404d00f-242b-4f97-a9c6-bb04c79d7348-kube-api-access-ngzr4\") pod \"payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl\" (UID: \"7404d00f-242b-4f97-a9c6-bb04c79d7348\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl"
Nov 03 00:31:07.711424 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:07.711303    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7404d00f-242b-4f97-a9c6-bb04c79d7348-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl\" (UID: \"7404d00f-242b-4f97-a9c6-bb04c79d7348\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl"
Nov 03 00:31:07.711988 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:07.711970    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7404d00f-242b-4f97-a9c6-bb04c79d7348-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl\" (UID: \"7404d00f-242b-4f97-a9c6-bb04c79d7348\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl"
Nov 03 00:31:07.728906 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:07.728873    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngzr4\" (UniqueName: \"kubernetes.io/projected/7404d00f-242b-4f97-a9c6-bb04c79d7348-kube-api-access-ngzr4\") pod \"payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl\" (UID: \"7404d00f-242b-4f97-a9c6-bb04c79d7348\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl"
Nov 03 00:31:07.887797 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:07.887773    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl"
Nov 03 00:31:08.188060 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:31:08.188029    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7404d00f_242b_4f97_a9c6_bb04c79d7348.slice/crio-6a374b609d127dd7d6fe643f5f825b19c29b127c1ac94dca907c634f7440395a WatchSource:0}: Error finding container 6a374b609d127dd7d6fe643f5f825b19c29b127c1ac94dca907c634f7440395a: Status 404 returned error can't find the container with id 6a374b609d127dd7d6fe643f5f825b19c29b127c1ac94dca907c634f7440395a
Nov 03 00:31:08.188436 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:08.188412    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl"]
Nov 03 00:31:08.730368 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:08.730324    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl" event={"ID":"7404d00f-242b-4f97-a9c6-bb04c79d7348","Type":"ContainerStarted","Data":"e69e0e74fd9097d07f19d4121be1246b5d6b4ed9db906480ac827bb324f77132"}
Nov 03 00:31:08.730820 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:08.730395    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl" event={"ID":"7404d00f-242b-4f97-a9c6-bb04c79d7348","Type":"ContainerStarted","Data":"6a374b609d127dd7d6fe643f5f825b19c29b127c1ac94dca907c634f7440395a"}
Nov 03 00:31:08.730820 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:08.730427    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl"
Nov 03 00:31:08.767308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:08.767232    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl" podStartSLOduration=1.767218412 podStartE2EDuration="1.767218412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:31:08.765437542 +0000 UTC m=+1551.645021584" watchObservedRunningTime="2026-11-03 00:31:08.767218412 +0000 UTC m=+1551.646802457"
Nov 03 00:31:08.844790 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:08.844760    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:31:08.859118 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:08.859092    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:31:09.750539 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:09.750510    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:31:19.351331 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:19.351298    5012 scope.go:122] "RemoveContainer" containerID="bebfb19ca5975e6198ca5e0f922d2342996098e8c36802fe3a67a08d65d0d698"
Nov 03 00:31:19.741137 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:19.741057    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl"
Nov 03 00:31:20.787022 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:20.786988    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"]
Nov 03 00:31:20.793229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:20.793204    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:31:20.814290 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:20.814261    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"]
Nov 03 00:31:20.815315 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:20.815290    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a43474\"/\"ipp-route-a43474-kserve-self-signed-certs\""
Nov 03 00:31:20.815655 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:20.815638    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a43474\"/\"default-dockercfg-6q9t8\""
Nov 03 00:31:20.816640 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:20.816618    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a43474\"/\"openshift-service-ca.crt\""
Nov 03 00:31:20.816748 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:20.816702    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a43474\"/\"kube-root-ca.crt\""
Nov 03 00:31:20.926714 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:20.926683    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-tmp-dir\") pod \"ipp-route-a43474-kserve-768f6487c-xtbkx\" (UID: \"10bb726f-ca52-4d99-994d-a82e852764df\") " pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:31:20.926856 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:20.926808    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kswtg\" (UniqueName: \"kubernetes.io/projected/10bb726f-ca52-4d99-994d-a82e852764df-kube-api-access-kswtg\") pod \"ipp-route-a43474-kserve-768f6487c-xtbkx\" (UID: \"10bb726f-ca52-4d99-994d-a82e852764df\") " pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:31:20.926856 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:20.926845    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-home\") pod \"ipp-route-a43474-kserve-768f6487c-xtbkx\" (UID: \"10bb726f-ca52-4d99-994d-a82e852764df\") " pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:31:20.926974 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:20.926903    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-model-cache\") pod \"ipp-route-a43474-kserve-768f6487c-xtbkx\" (UID: \"10bb726f-ca52-4d99-994d-a82e852764df\") " pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:31:20.927034 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:20.927020    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10bb726f-ca52-4d99-994d-a82e852764df-tls-certs\") pod \"ipp-route-a43474-kserve-768f6487c-xtbkx\" (UID: \"10bb726f-ca52-4d99-994d-a82e852764df\") " pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:31:20.927092 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:20.927077    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-dshm\") pod \"ipp-route-a43474-kserve-768f6487c-xtbkx\" (UID: \"10bb726f-ca52-4d99-994d-a82e852764df\") " pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:31:21.029771 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:21.029080    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kswtg\" (UniqueName: \"kubernetes.io/projected/10bb726f-ca52-4d99-994d-a82e852764df-kube-api-access-kswtg\") pod \"ipp-route-a43474-kserve-768f6487c-xtbkx\" (UID: \"10bb726f-ca52-4d99-994d-a82e852764df\") " pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:31:21.029771 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:21.029146    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-home\") pod \"ipp-route-a43474-kserve-768f6487c-xtbkx\" (UID: \"10bb726f-ca52-4d99-994d-a82e852764df\") " pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:31:21.029771 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:21.029182    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-model-cache\") pod \"ipp-route-a43474-kserve-768f6487c-xtbkx\" (UID: \"10bb726f-ca52-4d99-994d-a82e852764df\") " pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:31:21.029771 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:21.029270    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10bb726f-ca52-4d99-994d-a82e852764df-tls-certs\") pod \"ipp-route-a43474-kserve-768f6487c-xtbkx\" (UID: \"10bb726f-ca52-4d99-994d-a82e852764df\") " pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:31:21.029771 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:21.029326    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-dshm\") pod \"ipp-route-a43474-kserve-768f6487c-xtbkx\" (UID: \"10bb726f-ca52-4d99-994d-a82e852764df\") " pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:31:21.029771 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:21.029516    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-tmp-dir\") pod \"ipp-route-a43474-kserve-768f6487c-xtbkx\" (UID: \"10bb726f-ca52-4d99-994d-a82e852764df\") " pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:31:21.030271 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:21.030226    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-model-cache\") pod \"ipp-route-a43474-kserve-768f6487c-xtbkx\" (UID: \"10bb726f-ca52-4d99-994d-a82e852764df\") " pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:31:21.032269 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:21.030919    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-home\") pod \"ipp-route-a43474-kserve-768f6487c-xtbkx\" (UID: \"10bb726f-ca52-4d99-994d-a82e852764df\") " pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:31:21.032415 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:21.032391    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-tmp-dir\") pod \"ipp-route-a43474-kserve-768f6487c-xtbkx\" (UID: \"10bb726f-ca52-4d99-994d-a82e852764df\") " pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:31:21.046897 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:21.046797    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10bb726f-ca52-4d99-994d-a82e852764df-tls-certs\") pod \"ipp-route-a43474-kserve-768f6487c-xtbkx\" (UID: \"10bb726f-ca52-4d99-994d-a82e852764df\") " pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:31:21.046897 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:21.046860    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-dshm\") pod \"ipp-route-a43474-kserve-768f6487c-xtbkx\" (UID: \"10bb726f-ca52-4d99-994d-a82e852764df\") " pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:31:21.049524 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:21.049486    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kswtg\" (UniqueName: \"kubernetes.io/projected/10bb726f-ca52-4d99-994d-a82e852764df-kube-api-access-kswtg\") pod \"ipp-route-a43474-kserve-768f6487c-xtbkx\" (UID: \"10bb726f-ca52-4d99-994d-a82e852764df\") " pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:31:21.116374 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:21.116342    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:31:21.424504 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:31:21.424474    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10bb726f_ca52_4d99_994d_a82e852764df.slice/crio-2e935726580fc13b6c8e2682c98266eb7287531475ed75cea696e69332cad45f WatchSource:0}: Error finding container 2e935726580fc13b6c8e2682c98266eb7287531475ed75cea696e69332cad45f: Status 404 returned error can't find the container with id 2e935726580fc13b6c8e2682c98266eb7287531475ed75cea696e69332cad45f
Nov 03 00:31:21.424839 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:21.424815    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"]
Nov 03 00:31:21.853763 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:21.853725    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx" event={"ID":"10bb726f-ca52-4d99-994d-a82e852764df","Type":"ContainerStarted","Data":"745f93362bb36b735a58cecf8709c75e63c1e442c33453013c35a982cf30c457"}
Nov 03 00:31:21.854230 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:21.853778    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx" event={"ID":"10bb726f-ca52-4d99-994d-a82e852764df","Type":"ContainerStarted","Data":"2e935726580fc13b6c8e2682c98266eb7287531475ed75cea696e69332cad45f"}
Nov 03 00:31:21.893659 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:21.893610    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx" podStartSLOduration=1.893596982 podStartE2EDuration="1.893596982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:31:21.888448717 +0000 UTC m=+1564.768032760" watchObservedRunningTime="2026-11-03 00:31:21.893596982 +0000 UTC m=+1564.773181024"
Nov 03 00:31:22.117647 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:22.117534    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:31:31.116985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:31.116877    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:31:31.131300 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:31.131264    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:31:31.950040 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:31.950012    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:31:34.460802 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:34.460767    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7"]
Nov 03 00:31:34.461293 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:34.460996    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7" podUID="faff8e48-0925-4faf-84be-a5529573e717" containerName="payload-processing" containerID="cri-o://f1f605098d5fd45d22403730b0f7e3148ca93b0da755cd303e9bd2f559343a54" gracePeriod=30
Nov 03 00:31:34.494834 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:34.494793    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4"]
Nov 03 00:31:34.495089 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:34.495061    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4" podUID="933c9a50-87ce-4b10-ba38-3fcdd105b7f7" containerName="payload-pre-processing" containerID="cri-o://485e256f2acb887dfd2380386c79b28b685c743510d6cae68d58ebe3704b2d2a" gracePeriod=30
Nov 03 00:31:34.860856 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:34.860491    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7"
Nov 03 00:31:34.952887 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:34.952857    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faff8e48-0925-4faf-84be-a5529573e717-kube-api-access-t28px\" (UniqueName: \"kubernetes.io/projected/faff8e48-0925-4faf-84be-a5529573e717-kube-api-access-t28px\") pod \"faff8e48-0925-4faf-84be-a5529573e717\" (UID: \"faff8e48-0925-4faf-84be-a5529573e717\") "
Nov 03 00:31:34.953058 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:34.952927    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faff8e48-0925-4faf-84be-a5529573e717-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faff8e48-0925-4faf-84be-a5529573e717-plugins-config-volume\") pod \"faff8e48-0925-4faf-84be-a5529573e717\" (UID: \"faff8e48-0925-4faf-84be-a5529573e717\") "
Nov 03 00:31:34.955206 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:34.955177    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faff8e48-0925-4faf-84be-a5529573e717-kube-api-access-t28px" pod "faff8e48-0925-4faf-84be-a5529573e717" (UID: "faff8e48-0925-4faf-84be-a5529573e717"). InnerVolumeSpecName "kube-api-access-t28px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:31:34.956236 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:34.956210    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faff8e48-0925-4faf-84be-a5529573e717-plugins-config-volume" pod "faff8e48-0925-4faf-84be-a5529573e717" (UID: "faff8e48-0925-4faf-84be-a5529573e717"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:31:34.963826 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:34.963803    5012 generic.go:356] "Generic (PLEG): container finished" podID="933c9a50-87ce-4b10-ba38-3fcdd105b7f7" containerID="485e256f2acb887dfd2380386c79b28b685c743510d6cae68d58ebe3704b2d2a" exitCode=0
Nov 03 00:31:34.963960 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:34.963875    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4" event={"ID":"933c9a50-87ce-4b10-ba38-3fcdd105b7f7","Type":"ContainerDied","Data":"485e256f2acb887dfd2380386c79b28b685c743510d6cae68d58ebe3704b2d2a"}
Nov 03 00:31:34.963960 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:34.963950    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4" event={"ID":"933c9a50-87ce-4b10-ba38-3fcdd105b7f7","Type":"ContainerDied","Data":"4a319c8d7c9a648700299d678fdea0322a52aefb870cd098a178776244541f6e"}
Nov 03 00:31:34.964072 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:34.963976    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a319c8d7c9a648700299d678fdea0322a52aefb870cd098a178776244541f6e"
Nov 03 00:31:34.965635 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:34.965606    5012 generic.go:356] "Generic (PLEG): container finished" podID="faff8e48-0925-4faf-84be-a5529573e717" containerID="f1f605098d5fd45d22403730b0f7e3148ca93b0da755cd303e9bd2f559343a54" exitCode=0
Nov 03 00:31:34.965732 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:34.965664    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7"
Nov 03 00:31:34.965732 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:34.965694    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7" event={"ID":"faff8e48-0925-4faf-84be-a5529573e717","Type":"ContainerDied","Data":"f1f605098d5fd45d22403730b0f7e3148ca93b0da755cd303e9bd2f559343a54"}
Nov 03 00:31:34.965856 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:34.965746    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7" event={"ID":"faff8e48-0925-4faf-84be-a5529573e717","Type":"ContainerDied","Data":"203b95e3aff314ff088369e5dc0516e7db0b9bb99b32530c74a0b6432fdeed24"}
Nov 03 00:31:34.965856 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:34.965778    5012 scope.go:122] "RemoveContainer" containerID="f1f605098d5fd45d22403730b0f7e3148ca93b0da755cd303e9bd2f559343a54"
Nov 03 00:31:34.976433 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:34.976409    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4"
Nov 03 00:31:34.978327 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:34.978309    5012 scope.go:122] "RemoveContainer" containerID="f1f605098d5fd45d22403730b0f7e3148ca93b0da755cd303e9bd2f559343a54"
Nov 03 00:31:34.978902 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:31:34.978748    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1f605098d5fd45d22403730b0f7e3148ca93b0da755cd303e9bd2f559343a54\": container with ID starting with f1f605098d5fd45d22403730b0f7e3148ca93b0da755cd303e9bd2f559343a54 not found: ID does not exist" containerID="f1f605098d5fd45d22403730b0f7e3148ca93b0da755cd303e9bd2f559343a54"
Nov 03 00:31:34.978902 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:34.978776    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1f605098d5fd45d22403730b0f7e3148ca93b0da755cd303e9bd2f559343a54"} err="failed to get container status \"f1f605098d5fd45d22403730b0f7e3148ca93b0da755cd303e9bd2f559343a54\": rpc error: code = NotFound desc = could not find container \"f1f605098d5fd45d22403730b0f7e3148ca93b0da755cd303e9bd2f559343a54\": container with ID starting with f1f605098d5fd45d22403730b0f7e3148ca93b0da755cd303e9bd2f559343a54 not found: ID does not exist"
Nov 03 00:31:35.054624 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:35.054525    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/933c9a50-87ce-4b10-ba38-3fcdd105b7f7-kube-api-access-75lzn\" (UniqueName: \"kubernetes.io/projected/933c9a50-87ce-4b10-ba38-3fcdd105b7f7-kube-api-access-75lzn\") pod \"933c9a50-87ce-4b10-ba38-3fcdd105b7f7\" (UID: \"933c9a50-87ce-4b10-ba38-3fcdd105b7f7\") "
Nov 03 00:31:35.054805 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:35.054679    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/933c9a50-87ce-4b10-ba38-3fcdd105b7f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/933c9a50-87ce-4b10-ba38-3fcdd105b7f7-plugins-config-volume\") pod \"933c9a50-87ce-4b10-ba38-3fcdd105b7f7\" (UID: \"933c9a50-87ce-4b10-ba38-3fcdd105b7f7\") "
Nov 03 00:31:35.055344 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:35.054997    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7"]
Nov 03 00:31:35.055344 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:35.055136    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/933c9a50-87ce-4b10-ba38-3fcdd105b7f7-plugins-config-volume" pod "933c9a50-87ce-4b10-ba38-3fcdd105b7f7" (UID: "933c9a50-87ce-4b10-ba38-3fcdd105b7f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:31:35.055344 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:35.055282    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t28px\" (UniqueName: \"kubernetes.io/projected/faff8e48-0925-4faf-84be-a5529573e717-kube-api-access-t28px\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:31:35.055344 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:35.055305    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faff8e48-0925-4faf-84be-a5529573e717-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:31:35.055344 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:35.055320    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/933c9a50-87ce-4b10-ba38-3fcdd105b7f7-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:31:35.057206 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:35.057184    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933c9a50-87ce-4b10-ba38-3fcdd105b7f7-kube-api-access-75lzn" pod "933c9a50-87ce-4b10-ba38-3fcdd105b7f7" (UID: "933c9a50-87ce-4b10-ba38-3fcdd105b7f7"). InnerVolumeSpecName "kube-api-access-75lzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:31:35.060182 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:35.060163    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-54f6d8-694bdf8c64-2lth7"]
Nov 03 00:31:35.156901 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:35.156877    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75lzn\" (UniqueName: \"kubernetes.io/projected/933c9a50-87ce-4b10-ba38-3fcdd105b7f7-kube-api-access-75lzn\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:31:35.245443 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:35.245414    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faff8e48-0925-4faf-84be-a5529573e717" path="/var/lib/kubelet/pods/faff8e48-0925-4faf-84be-a5529573e717/volumes"
Nov 03 00:31:35.974627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:35.974596    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4"
Nov 03 00:31:36.021688 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:36.021660    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4"]
Nov 03 00:31:36.031921 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:36.031882    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-54f6d8-f47b98d7f-rmwd4"]
Nov 03 00:31:37.247481 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:37.247449    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="933c9a50-87ce-4b10-ba38-3fcdd105b7f7" path="/var/lib/kubelet/pods/933c9a50-87ce-4b10-ba38-3fcdd105b7f7/volumes"
Nov 03 00:31:37.363395 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:31:37.363357    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:31:39.408627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.408584    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg"]
Nov 03 00:31:39.409786 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.409757    5012 cpu_manager.go:414] "containerMap: removing container" podUID="933c9a50-87ce-4b10-ba38-3fcdd105b7f7" containerName="payload-pre-processing"
Nov 03 00:31:39.409786 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.409786    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="933c9a50-87ce-4b10-ba38-3fcdd105b7f7" containerName="payload-pre-processing"
Nov 03 00:31:39.410014 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.409813    5012 cpu_manager.go:414] "containerMap: removing container" podUID="faff8e48-0925-4faf-84be-a5529573e717" containerName="payload-processing"
Nov 03 00:31:39.410014 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.409821    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faff8e48-0925-4faf-84be-a5529573e717" containerName="payload-processing"
Nov 03 00:31:39.410128 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.410051    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="933c9a50-87ce-4b10-ba38-3fcdd105b7f7" containerName="payload-pre-processing"
Nov 03 00:31:39.410128 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.410067    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="faff8e48-0925-4faf-84be-a5529573e717" containerName="payload-processing"
Nov 03 00:31:39.414765 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.414742    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg"
Nov 03 00:31:39.440929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.439743    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b6cd31\""
Nov 03 00:31:39.441793 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.441764    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b6cd31-dockercfg-v9pqb\""
Nov 03 00:31:39.491850 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.481394    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg"]
Nov 03 00:31:39.513977 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.513936    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac19255-340c-4891-85d4-1b895e2d3641-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg\" (UID: \"9ac19255-340c-4891-85d4-1b895e2d3641\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg"
Nov 03 00:31:39.514162 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.514053    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85456\" (UniqueName: \"kubernetes.io/projected/9ac19255-340c-4891-85d4-1b895e2d3641-kube-api-access-85456\") pod \"payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg\" (UID: \"9ac19255-340c-4891-85d4-1b895e2d3641\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg"
Nov 03 00:31:39.578353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.578297    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g"]
Nov 03 00:31:39.582877 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.582856    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g"
Nov 03 00:31:39.601646 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.601624    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g"]
Nov 03 00:31:39.615036 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.615010    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac19255-340c-4891-85d4-1b895e2d3641-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg\" (UID: \"9ac19255-340c-4891-85d4-1b895e2d3641\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg"
Nov 03 00:31:39.615167 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.615106    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85456\" (UniqueName: \"kubernetes.io/projected/9ac19255-340c-4891-85d4-1b895e2d3641-kube-api-access-85456\") pod \"payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg\" (UID: \"9ac19255-340c-4891-85d4-1b895e2d3641\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg"
Nov 03 00:31:39.615771 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.615748    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac19255-340c-4891-85d4-1b895e2d3641-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg\" (UID: \"9ac19255-340c-4891-85d4-1b895e2d3641\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg"
Nov 03 00:31:39.632541 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.632517    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85456\" (UniqueName: \"kubernetes.io/projected/9ac19255-340c-4891-85d4-1b895e2d3641-kube-api-access-85456\") pod \"payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg\" (UID: \"9ac19255-340c-4891-85d4-1b895e2d3641\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg"
Nov 03 00:31:39.716049 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.715993    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5689r\" (UniqueName: \"kubernetes.io/projected/b2570d35-c9ee-4069-85c2-361cba7c27fb-kube-api-access-5689r\") pod \"payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g\" (UID: \"b2570d35-c9ee-4069-85c2-361cba7c27fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g"
Nov 03 00:31:39.716168 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.716154    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2570d35-c9ee-4069-85c2-361cba7c27fb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g\" (UID: \"b2570d35-c9ee-4069-85c2-361cba7c27fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g"
Nov 03 00:31:39.739532 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.739507    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg"
Nov 03 00:31:39.822475 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.820851    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5689r\" (UniqueName: \"kubernetes.io/projected/b2570d35-c9ee-4069-85c2-361cba7c27fb-kube-api-access-5689r\") pod \"payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g\" (UID: \"b2570d35-c9ee-4069-85c2-361cba7c27fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g"
Nov 03 00:31:39.822475 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.821138    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2570d35-c9ee-4069-85c2-361cba7c27fb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g\" (UID: \"b2570d35-c9ee-4069-85c2-361cba7c27fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g"
Nov 03 00:31:39.822475 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.822329    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2570d35-c9ee-4069-85c2-361cba7c27fb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g\" (UID: \"b2570d35-c9ee-4069-85c2-361cba7c27fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g"
Nov 03 00:31:39.840957 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.839739    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5689r\" (UniqueName: \"kubernetes.io/projected/b2570d35-c9ee-4069-85c2-361cba7c27fb-kube-api-access-5689r\") pod \"payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g\" (UID: \"b2570d35-c9ee-4069-85c2-361cba7c27fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g"
Nov 03 00:31:39.898539 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:39.898511    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g"
Nov 03 00:31:40.043817 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:31:40.043787    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac19255_340c_4891_85d4_1b895e2d3641.slice/crio-fffd7d60ba3dc921efc3614c3c0b9306a950f6e18de3d3d3761db6ac4074d441 WatchSource:0}: Error finding container fffd7d60ba3dc921efc3614c3c0b9306a950f6e18de3d3d3761db6ac4074d441: Status 404 returned error can't find the container with id fffd7d60ba3dc921efc3614c3c0b9306a950f6e18de3d3d3761db6ac4074d441
Nov 03 00:31:40.045516 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:40.045483    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg"]
Nov 03 00:31:40.204742 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:40.204717    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g"]
Nov 03 00:31:40.204742 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:31:40.204729    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2570d35_c9ee_4069_85c2_361cba7c27fb.slice/crio-a6f27e94ef171e80b06a23fbfeb2c0fcfac0c698e43c0b87615ff3c92f917fef WatchSource:0}: Error finding container a6f27e94ef171e80b06a23fbfeb2c0fcfac0c698e43c0b87615ff3c92f917fef: Status 404 returned error can't find the container with id a6f27e94ef171e80b06a23fbfeb2c0fcfac0c698e43c0b87615ff3c92f917fef
Nov 03 00:31:41.023333 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:41.023295    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg" event={"ID":"9ac19255-340c-4891-85d4-1b895e2d3641","Type":"ContainerStarted","Data":"343f4a9bb285af96e3e41b3b2a8f4d77688ece2ca0eb55081163e8da3e3ecd34"}
Nov 03 00:31:41.023822 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:41.023367    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg"
Nov 03 00:31:41.023822 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:41.023392    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg" event={"ID":"9ac19255-340c-4891-85d4-1b895e2d3641","Type":"ContainerStarted","Data":"fffd7d60ba3dc921efc3614c3c0b9306a950f6e18de3d3d3761db6ac4074d441"}
Nov 03 00:31:41.025376 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:41.025337    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g" event={"ID":"b2570d35-c9ee-4069-85c2-361cba7c27fb","Type":"ContainerStarted","Data":"d6c7292bb52a34cad1f20904fb7763db99739a7eef591c1c31f807dd3a976b0c"}
Nov 03 00:31:41.025528 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:41.025399    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g" event={"ID":"b2570d35-c9ee-4069-85c2-361cba7c27fb","Type":"ContainerStarted","Data":"a6f27e94ef171e80b06a23fbfeb2c0fcfac0c698e43c0b87615ff3c92f917fef"}
Nov 03 00:31:41.025528 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:41.025427    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g"
Nov 03 00:31:41.060622 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:41.060575    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg" podStartSLOduration=2.060560837 podStartE2EDuration="2.060560837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:31:41.056826748 +0000 UTC m=+1583.936410788" watchObservedRunningTime="2026-11-03 00:31:41.060560837 +0000 UTC m=+1583.940144879"
Nov 03 00:31:41.096932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:41.095640    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g" podStartSLOduration=2.095622032 podStartE2EDuration="2.095622032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:31:41.090848715 +0000 UTC m=+1583.970432759" watchObservedRunningTime="2026-11-03 00:31:41.095622032 +0000 UTC m=+1583.975206077"
Nov 03 00:31:52.036341 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:52.036311    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg"
Nov 03 00:31:52.036846 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:31:52.036385    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g"
Nov 03 00:32:04.377064 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:04.377032    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx"]
Nov 03 00:32:04.377553 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:04.377215    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx" podUID="a1d8aac2-8d32-4e6f-a342-d2e2c7425f18" containerName="maas-api" containerID="cri-o://f03ee31151a91607cad779964acbaaadb4c22e193489b66051992944c4aa055f" gracePeriod=30
Nov 03 00:32:04.570131 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:04.569958    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl"]
Nov 03 00:32:04.570316 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:04.570178    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl" podUID="7404d00f-242b-4f97-a9c6-bb04c79d7348" containerName="payload-pre-processing" containerID="cri-o://e69e0e74fd9097d07f19d4121be1246b5d6b4ed9db906480ac827bb324f77132" gracePeriod=30
Nov 03 00:32:04.707495 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:04.707467    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx"
Nov 03 00:32:04.834642 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:04.834612    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-openshift-service-ca\") pod \"a1d8aac2-8d32-4e6f-a342-d2e2c7425f18\" (UID: \"a1d8aac2-8d32-4e6f-a342-d2e2c7425f18\") "
Nov 03 00:32:04.834829 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:04.834655    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-metrics-certs\") pod \"a1d8aac2-8d32-4e6f-a342-d2e2c7425f18\" (UID: \"a1d8aac2-8d32-4e6f-a342-d2e2c7425f18\") "
Nov 03 00:32:04.834829 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:04.834683    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-kube-api-access-cdw55\" (UniqueName: \"kubernetes.io/projected/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-kube-api-access-cdw55\") pod \"a1d8aac2-8d32-4e6f-a342-d2e2c7425f18\" (UID: \"a1d8aac2-8d32-4e6f-a342-d2e2c7425f18\") "
Nov 03 00:32:04.834829 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:04.834705    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-maas-api-tls\") pod \"a1d8aac2-8d32-4e6f-a342-d2e2c7425f18\" (UID: \"a1d8aac2-8d32-4e6f-a342-d2e2c7425f18\") "
Nov 03 00:32:04.835115 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:04.835090    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-openshift-service-ca" pod "a1d8aac2-8d32-4e6f-a342-d2e2c7425f18" (UID: "a1d8aac2-8d32-4e6f-a342-d2e2c7425f18"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:04.835443 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:04.835413    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:04.836878 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:04.836849    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-maas-api-tls" pod "a1d8aac2-8d32-4e6f-a342-d2e2c7425f18" (UID: "a1d8aac2-8d32-4e6f-a342-d2e2c7425f18"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:32:04.837038 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:04.837009    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-kube-api-access-cdw55" pod "a1d8aac2-8d32-4e6f-a342-d2e2c7425f18" (UID: "a1d8aac2-8d32-4e6f-a342-d2e2c7425f18"). InnerVolumeSpecName "kube-api-access-cdw55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:04.837104 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:04.837087    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-metrics-certs" pod "a1d8aac2-8d32-4e6f-a342-d2e2c7425f18" (UID: "a1d8aac2-8d32-4e6f-a342-d2e2c7425f18"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:32:04.908719 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:04.908683    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl"
Nov 03 00:32:04.945990 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:04.938373    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:04.945990 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:04.938411    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdw55\" (UniqueName: \"kubernetes.io/projected/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-kube-api-access-cdw55\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:04.945990 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:04.938437    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:05.039689 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.039655    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7404d00f-242b-4f97-a9c6-bb04c79d7348-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7404d00f-242b-4f97-a9c6-bb04c79d7348-plugins-config-volume\") pod \"7404d00f-242b-4f97-a9c6-bb04c79d7348\" (UID: \"7404d00f-242b-4f97-a9c6-bb04c79d7348\") "
Nov 03 00:32:05.039896 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.039776    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7404d00f-242b-4f97-a9c6-bb04c79d7348-kube-api-access-ngzr4\" (UniqueName: \"kubernetes.io/projected/7404d00f-242b-4f97-a9c6-bb04c79d7348-kube-api-access-ngzr4\") pod \"7404d00f-242b-4f97-a9c6-bb04c79d7348\" (UID: \"7404d00f-242b-4f97-a9c6-bb04c79d7348\") "
Nov 03 00:32:05.040171 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.040144    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7404d00f-242b-4f97-a9c6-bb04c79d7348-plugins-config-volume" pod "7404d00f-242b-4f97-a9c6-bb04c79d7348" (UID: "7404d00f-242b-4f97-a9c6-bb04c79d7348"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:05.040285 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.040271    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7404d00f-242b-4f97-a9c6-bb04c79d7348-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:05.041972 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.041905    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7404d00f-242b-4f97-a9c6-bb04c79d7348-kube-api-access-ngzr4" pod "7404d00f-242b-4f97-a9c6-bb04c79d7348" (UID: "7404d00f-242b-4f97-a9c6-bb04c79d7348"). InnerVolumeSpecName "kube-api-access-ngzr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:05.141826 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.141794    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngzr4\" (UniqueName: \"kubernetes.io/projected/7404d00f-242b-4f97-a9c6-bb04c79d7348-kube-api-access-ngzr4\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:05.239819 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.239748    5012 generic.go:356] "Generic (PLEG): container finished" podID="7404d00f-242b-4f97-a9c6-bb04c79d7348" containerID="e69e0e74fd9097d07f19d4121be1246b5d6b4ed9db906480ac827bb324f77132" exitCode=0
Nov 03 00:32:05.239964 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.239848    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl"
Nov 03 00:32:05.241487 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.241462    5012 generic.go:356] "Generic (PLEG): container finished" podID="a1d8aac2-8d32-4e6f-a342-d2e2c7425f18" containerID="f03ee31151a91607cad779964acbaaadb4c22e193489b66051992944c4aa055f" exitCode=0
Nov 03 00:32:05.241632 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.241573    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx"
Nov 03 00:32:05.248102 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.248063    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl" event={"ID":"7404d00f-242b-4f97-a9c6-bb04c79d7348","Type":"ContainerDied","Data":"e69e0e74fd9097d07f19d4121be1246b5d6b4ed9db906480ac827bb324f77132"}
Nov 03 00:32:05.248247 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.248125    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl" event={"ID":"7404d00f-242b-4f97-a9c6-bb04c79d7348","Type":"ContainerDied","Data":"6a374b609d127dd7d6fe643f5f825b19c29b127c1ac94dca907c634f7440395a"}
Nov 03 00:32:05.248247 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.248149    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx" event={"ID":"a1d8aac2-8d32-4e6f-a342-d2e2c7425f18","Type":"ContainerDied","Data":"f03ee31151a91607cad779964acbaaadb4c22e193489b66051992944c4aa055f"}
Nov 03 00:32:05.248247 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.248171    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx" event={"ID":"a1d8aac2-8d32-4e6f-a342-d2e2c7425f18","Type":"ContainerDied","Data":"317cce3a169db5cb65aa1e55fe3a1afcf7416189205917f2158370f0c9aa006d"}
Nov 03 00:32:05.248247 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.248201    5012 scope.go:122] "RemoveContainer" containerID="e69e0e74fd9097d07f19d4121be1246b5d6b4ed9db906480ac827bb324f77132"
Nov 03 00:32:05.265294 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.265274    5012 scope.go:122] "RemoveContainer" containerID="e69e0e74fd9097d07f19d4121be1246b5d6b4ed9db906480ac827bb324f77132"
Nov 03 00:32:05.265577 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:32:05.265556    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e69e0e74fd9097d07f19d4121be1246b5d6b4ed9db906480ac827bb324f77132\": container with ID starting with e69e0e74fd9097d07f19d4121be1246b5d6b4ed9db906480ac827bb324f77132 not found: ID does not exist" containerID="e69e0e74fd9097d07f19d4121be1246b5d6b4ed9db906480ac827bb324f77132"
Nov 03 00:32:05.265686 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.265584    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e69e0e74fd9097d07f19d4121be1246b5d6b4ed9db906480ac827bb324f77132"} err="failed to get container status \"e69e0e74fd9097d07f19d4121be1246b5d6b4ed9db906480ac827bb324f77132\": rpc error: code = NotFound desc = could not find container \"e69e0e74fd9097d07f19d4121be1246b5d6b4ed9db906480ac827bb324f77132\": container with ID starting with e69e0e74fd9097d07f19d4121be1246b5d6b4ed9db906480ac827bb324f77132 not found: ID does not exist"
Nov 03 00:32:05.265686 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.265606    5012 scope.go:122] "RemoveContainer" containerID="f03ee31151a91607cad779964acbaaadb4c22e193489b66051992944c4aa055f"
Nov 03 00:32:05.278345 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.278322    5012 scope.go:122] "RemoveContainer" containerID="f03ee31151a91607cad779964acbaaadb4c22e193489b66051992944c4aa055f"
Nov 03 00:32:05.278619 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:32:05.278595    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f03ee31151a91607cad779964acbaaadb4c22e193489b66051992944c4aa055f\": container with ID starting with f03ee31151a91607cad779964acbaaadb4c22e193489b66051992944c4aa055f not found: ID does not exist" containerID="f03ee31151a91607cad779964acbaaadb4c22e193489b66051992944c4aa055f"
Nov 03 00:32:05.278682 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.278626    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f03ee31151a91607cad779964acbaaadb4c22e193489b66051992944c4aa055f"} err="failed to get container status \"f03ee31151a91607cad779964acbaaadb4c22e193489b66051992944c4aa055f\": rpc error: code = NotFound desc = could not find container \"f03ee31151a91607cad779964acbaaadb4c22e193489b66051992944c4aa055f\": container with ID starting with f03ee31151a91607cad779964acbaaadb4c22e193489b66051992944c4aa055f not found: ID does not exist"
Nov 03 00:32:05.297613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.297591    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx"]
Nov 03 00:32:05.302026 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.302007    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea275f-6969ccb5f9-8kvhx"]
Nov 03 00:32:05.305533 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.305512    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"]
Nov 03 00:32:05.305710 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.305692    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd" podUID="8760ca6d-230b-4d95-ae34-d9384b441607" containerName="istio-proxy" containerID="cri-o://193de80f706d976677f9e6fe90c33184d014cb5dbe1f3a6691ffa63646e3da05" gracePeriod=30
Nov 03 00:32:05.336181 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.336157    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl"]
Nov 03 00:32:05.344517 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.344495    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea275f-55b77f9d7f-t6wdl"]
Nov 03 00:32:05.357308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.357283    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"]
Nov 03 00:32:05.357519 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:05.357474    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd" podUID="2f8b0c71-4f60-47e3-9a22-7f25e85b690c" containerName="istio-proxy" containerID="cri-o://ca6073d57aee0e6959f3d0ac717d95d3dac931794354b30bde191282a161ccda" gracePeriod=30
Nov 03 00:32:07.247109 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:07.247073    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7404d00f-242b-4f97-a9c6-bb04c79d7348" path="/var/lib/kubelet/pods/7404d00f-242b-4f97-a9c6-bb04c79d7348/volumes"
Nov 03 00:32:07.247518 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:07.247503    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1d8aac2-8d32-4e6f-a342-d2e2c7425f18" path="/var/lib/kubelet/pods/a1d8aac2-8d32-4e6f-a342-d2e2c7425f18/volumes"
Nov 03 00:32:07.399216 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:32:07.399182    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:32:09.672244 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:09.672208    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm"]
Nov 03 00:32:09.672746 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:09.672408    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm" podUID="0528abca-dac8-491c-b6ab-d183adf6bf8a" containerName="payload-processing" containerID="cri-o://6aefa2f8e7e7b12322d1396a7f667ed32325159a525042217deb7b202581b301" gracePeriod=30
Nov 03 00:32:09.701290 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:09.701262    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff"]
Nov 03 00:32:09.701604 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:09.701562    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff" podUID="72825023-1f03-47a3-a213-4cf1656e108a" containerName="payload-pre-processing" containerID="cri-o://edabaaa2ef1bd3c814da594d5207af5384aa6248be2e71ab3f96b7e0b4ee095b" gracePeriod=30
Nov 03 00:32:10.143654 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.143619    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm"
Nov 03 00:32:10.145759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.145681    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff"
Nov 03 00:32:10.242985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.232047    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0528abca-dac8-491c-b6ab-d183adf6bf8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0528abca-dac8-491c-b6ab-d183adf6bf8a-plugins-config-volume\") pod \"0528abca-dac8-491c-b6ab-d183adf6bf8a\" (UID: \"0528abca-dac8-491c-b6ab-d183adf6bf8a\") "
Nov 03 00:32:10.242985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.232096    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72825023-1f03-47a3-a213-4cf1656e108a-kube-api-access-xdhbv\" (UniqueName: \"kubernetes.io/projected/72825023-1f03-47a3-a213-4cf1656e108a-kube-api-access-xdhbv\") pod \"72825023-1f03-47a3-a213-4cf1656e108a\" (UID: \"72825023-1f03-47a3-a213-4cf1656e108a\") "
Nov 03 00:32:10.242985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.232375    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72825023-1f03-47a3-a213-4cf1656e108a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72825023-1f03-47a3-a213-4cf1656e108a-plugins-config-volume\") pod \"72825023-1f03-47a3-a213-4cf1656e108a\" (UID: \"72825023-1f03-47a3-a213-4cf1656e108a\") "
Nov 03 00:32:10.242985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.232500    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0528abca-dac8-491c-b6ab-d183adf6bf8a-kube-api-access-t2wfw\" (UniqueName: \"kubernetes.io/projected/0528abca-dac8-491c-b6ab-d183adf6bf8a-kube-api-access-t2wfw\") pod \"0528abca-dac8-491c-b6ab-d183adf6bf8a\" (UID: \"0528abca-dac8-491c-b6ab-d183adf6bf8a\") "
Nov 03 00:32:10.242985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.234002    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0528abca-dac8-491c-b6ab-d183adf6bf8a-plugins-config-volume" pod "0528abca-dac8-491c-b6ab-d183adf6bf8a" (UID: "0528abca-dac8-491c-b6ab-d183adf6bf8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:10.242985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.234482    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72825023-1f03-47a3-a213-4cf1656e108a-plugins-config-volume" pod "72825023-1f03-47a3-a213-4cf1656e108a" (UID: "72825023-1f03-47a3-a213-4cf1656e108a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:10.251171 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.251133    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0528abca-dac8-491c-b6ab-d183adf6bf8a-kube-api-access-t2wfw" pod "0528abca-dac8-491c-b6ab-d183adf6bf8a" (UID: "0528abca-dac8-491c-b6ab-d183adf6bf8a"). InnerVolumeSpecName "kube-api-access-t2wfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:10.261966 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.258439    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72825023-1f03-47a3-a213-4cf1656e108a-kube-api-access-xdhbv" pod "72825023-1f03-47a3-a213-4cf1656e108a" (UID: "72825023-1f03-47a3-a213-4cf1656e108a"). InnerVolumeSpecName "kube-api-access-xdhbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:10.334537 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.334502    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2wfw\" (UniqueName: \"kubernetes.io/projected/0528abca-dac8-491c-b6ab-d183adf6bf8a-kube-api-access-t2wfw\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.334681 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.334541    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0528abca-dac8-491c-b6ab-d183adf6bf8a-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.334681 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.334556    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdhbv\" (UniqueName: \"kubernetes.io/projected/72825023-1f03-47a3-a213-4cf1656e108a-kube-api-access-xdhbv\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.334681 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.334569    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72825023-1f03-47a3-a213-4cf1656e108a-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.346138 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.346098    5012 generic.go:356] "Generic (PLEG): container finished" podID="0528abca-dac8-491c-b6ab-d183adf6bf8a" containerID="6aefa2f8e7e7b12322d1396a7f667ed32325159a525042217deb7b202581b301" exitCode=0
Nov 03 00:32:10.346264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.346139    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm"
Nov 03 00:32:10.346264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.346168    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm" event={"ID":"0528abca-dac8-491c-b6ab-d183adf6bf8a","Type":"ContainerDied","Data":"6aefa2f8e7e7b12322d1396a7f667ed32325159a525042217deb7b202581b301"}
Nov 03 00:32:10.346264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.346219    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm" event={"ID":"0528abca-dac8-491c-b6ab-d183adf6bf8a","Type":"ContainerDied","Data":"52ebed9f43315f3ab69c205e6f3f85be6c61441d8bc47a1da8e69824a550e5d1"}
Nov 03 00:32:10.346264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.346237    5012 scope.go:122] "RemoveContainer" containerID="6aefa2f8e7e7b12322d1396a7f667ed32325159a525042217deb7b202581b301"
Nov 03 00:32:10.347730 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.347709    5012 generic.go:356] "Generic (PLEG): container finished" podID="72825023-1f03-47a3-a213-4cf1656e108a" containerID="edabaaa2ef1bd3c814da594d5207af5384aa6248be2e71ab3f96b7e0b4ee095b" exitCode=0
Nov 03 00:32:10.347837 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.347767    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff"
Nov 03 00:32:10.347837 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.347803    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff" event={"ID":"72825023-1f03-47a3-a213-4cf1656e108a","Type":"ContainerDied","Data":"edabaaa2ef1bd3c814da594d5207af5384aa6248be2e71ab3f96b7e0b4ee095b"}
Nov 03 00:32:10.347991 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.347846    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff" event={"ID":"72825023-1f03-47a3-a213-4cf1656e108a","Type":"ContainerDied","Data":"67cb46bc2550fb495d18fed80717b418a38d05fca6d024957860140c44ce596d"}
Nov 03 00:32:10.406996 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.406962    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff"]
Nov 03 00:32:10.411052 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.411026    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-75d94d47c9-cmvff"]
Nov 03 00:32:10.431072 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.431050    5012 scope.go:122] "RemoveContainer" containerID="6aefa2f8e7e7b12322d1396a7f667ed32325159a525042217deb7b202581b301"
Nov 03 00:32:10.431422 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:32:10.431400    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aefa2f8e7e7b12322d1396a7f667ed32325159a525042217deb7b202581b301\": container with ID starting with 6aefa2f8e7e7b12322d1396a7f667ed32325159a525042217deb7b202581b301 not found: ID does not exist" containerID="6aefa2f8e7e7b12322d1396a7f667ed32325159a525042217deb7b202581b301"
Nov 03 00:32:10.431481 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.431434    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aefa2f8e7e7b12322d1396a7f667ed32325159a525042217deb7b202581b301"} err="failed to get container status \"6aefa2f8e7e7b12322d1396a7f667ed32325159a525042217deb7b202581b301\": rpc error: code = NotFound desc = could not find container \"6aefa2f8e7e7b12322d1396a7f667ed32325159a525042217deb7b202581b301\": container with ID starting with 6aefa2f8e7e7b12322d1396a7f667ed32325159a525042217deb7b202581b301 not found: ID does not exist"
Nov 03 00:32:10.431481 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.431455    5012 scope.go:122] "RemoveContainer" containerID="edabaaa2ef1bd3c814da594d5207af5384aa6248be2e71ab3f96b7e0b4ee095b"
Nov 03 00:32:10.440585 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.440559    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm"]
Nov 03 00:32:10.445128 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.445103    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-846492-6b597c56c6-fsjmm"]
Nov 03 00:32:10.475247 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.475222    5012 scope.go:122] "RemoveContainer" containerID="edabaaa2ef1bd3c814da594d5207af5384aa6248be2e71ab3f96b7e0b4ee095b"
Nov 03 00:32:10.475631 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:32:10.475603    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edabaaa2ef1bd3c814da594d5207af5384aa6248be2e71ab3f96b7e0b4ee095b\": container with ID starting with edabaaa2ef1bd3c814da594d5207af5384aa6248be2e71ab3f96b7e0b4ee095b not found: ID does not exist" containerID="edabaaa2ef1bd3c814da594d5207af5384aa6248be2e71ab3f96b7e0b4ee095b"
Nov 03 00:32:10.475742 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.475646    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edabaaa2ef1bd3c814da594d5207af5384aa6248be2e71ab3f96b7e0b4ee095b"} err="failed to get container status \"edabaaa2ef1bd3c814da594d5207af5384aa6248be2e71ab3f96b7e0b4ee095b\": rpc error: code = NotFound desc = could not find container \"edabaaa2ef1bd3c814da594d5207af5384aa6248be2e71ab3f96b7e0b4ee095b\": container with ID starting with edabaaa2ef1bd3c814da594d5207af5384aa6248be2e71ab3f96b7e0b4ee095b not found: ID does not exist"
Nov 03 00:32:10.644970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.644938    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:32:10.755275 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.740445    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-credential-socket\") pod \"8760ca6d-230b-4d95-ae34-d9384b441607\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") "
Nov 03 00:32:10.755275 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.740490    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8760ca6d-230b-4d95-ae34-d9384b441607-istio-token\" (UniqueName: \"kubernetes.io/projected/8760ca6d-230b-4d95-ae34-d9384b441607-istio-token\") pod \"8760ca6d-230b-4d95-ae34-d9384b441607\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") "
Nov 03 00:32:10.755275 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.740555    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-istio-data\") pod \"8760ca6d-230b-4d95-ae34-d9384b441607\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") "
Nov 03 00:32:10.755275 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.740609    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8760ca6d-230b-4d95-ae34-d9384b441607-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8760ca6d-230b-4d95-ae34-d9384b441607-istio-podinfo\") pod \"8760ca6d-230b-4d95-ae34-d9384b441607\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") "
Nov 03 00:32:10.755275 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.740639    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-workload-certs\") pod \"8760ca6d-230b-4d95-ae34-d9384b441607\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") "
Nov 03 00:32:10.755275 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.740806    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-workload-socket\") pod \"8760ca6d-230b-4d95-ae34-d9384b441607\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") "
Nov 03 00:32:10.755275 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.740921    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-istio-envoy\") pod \"8760ca6d-230b-4d95-ae34-d9384b441607\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") "
Nov 03 00:32:10.755275 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.740959    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8760ca6d-230b-4d95-ae34-d9384b441607-kube-api-access-62m2j\" (UniqueName: \"kubernetes.io/projected/8760ca6d-230b-4d95-ae34-d9384b441607-kube-api-access-62m2j\") pod \"8760ca6d-230b-4d95-ae34-d9384b441607\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") "
Nov 03 00:32:10.755275 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.741038    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8760ca6d-230b-4d95-ae34-d9384b441607-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8760ca6d-230b-4d95-ae34-d9384b441607-istiod-ca-cert\") pod \"8760ca6d-230b-4d95-ae34-d9384b441607\" (UID: \"8760ca6d-230b-4d95-ae34-d9384b441607\") "
Nov 03 00:32:10.755275 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.751478    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8760ca6d-230b-4d95-ae34-d9384b441607-istiod-ca-cert" pod "8760ca6d-230b-4d95-ae34-d9384b441607" (UID: "8760ca6d-230b-4d95-ae34-d9384b441607"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:10.755275 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.751895    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-workload-certs" pod "8760ca6d-230b-4d95-ae34-d9384b441607" (UID: "8760ca6d-230b-4d95-ae34-d9384b441607"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.755275 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.752202    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-istio-data" pod "8760ca6d-230b-4d95-ae34-d9384b441607" (UID: "8760ca6d-230b-4d95-ae34-d9384b441607"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.755275 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.752394    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-credential-socket" pod "8760ca6d-230b-4d95-ae34-d9384b441607" (UID: "8760ca6d-230b-4d95-ae34-d9384b441607"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.755275 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.752459    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-workload-socket" pod "8760ca6d-230b-4d95-ae34-d9384b441607" (UID: "8760ca6d-230b-4d95-ae34-d9384b441607"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.758390 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.758359    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8760ca6d-230b-4d95-ae34-d9384b441607-istio-token" pod "8760ca6d-230b-4d95-ae34-d9384b441607" (UID: "8760ca6d-230b-4d95-ae34-d9384b441607"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:10.758795 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.758771    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8760ca6d-230b-4d95-ae34-d9384b441607-kube-api-access-62m2j" pod "8760ca6d-230b-4d95-ae34-d9384b441607" (UID: "8760ca6d-230b-4d95-ae34-d9384b441607"). InnerVolumeSpecName "kube-api-access-62m2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:10.759151 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.759126    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-istio-envoy" pod "8760ca6d-230b-4d95-ae34-d9384b441607" (UID: "8760ca6d-230b-4d95-ae34-d9384b441607"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.763461 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.763434    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8760ca6d-230b-4d95-ae34-d9384b441607-istio-podinfo" pod "8760ca6d-230b-4d95-ae34-d9384b441607" (UID: "8760ca6d-230b-4d95-ae34-d9384b441607"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:32:10.795471 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.795408    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:32:10.857360 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.857007    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.857360 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.857060    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62m2j\" (UniqueName: \"kubernetes.io/projected/8760ca6d-230b-4d95-ae34-d9384b441607-kube-api-access-62m2j\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.857360 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.857075    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8760ca6d-230b-4d95-ae34-d9384b441607-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.857360 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.857089    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.857360 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.857103    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8760ca6d-230b-4d95-ae34-d9384b441607-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.857360 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.857116    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.857360 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.857128    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8760ca6d-230b-4d95-ae34-d9384b441607-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.857360 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.857141    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.857360 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.857154    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8760ca6d-230b-4d95-ae34-d9384b441607-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.957677 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.957624    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-podinfo\") pod \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") "
Nov 03 00:32:10.958184 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.958153    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-token\" (UniqueName: \"kubernetes.io/projected/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-token\") pod \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") "
Nov 03 00:32:10.958348 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.958205    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-workload-certs\") pod \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") "
Nov 03 00:32:10.958348 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.958316    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-workload-socket\") pod \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") "
Nov 03 00:32:10.958487 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.958359    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-kube-api-access-tx9f9\" (UniqueName: \"kubernetes.io/projected/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-kube-api-access-tx9f9\") pod \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") "
Nov 03 00:32:10.958487 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.958437    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istiod-ca-cert\") pod \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") "
Nov 03 00:32:10.958608 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.958529    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-credential-socket\") pod \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") "
Nov 03 00:32:10.958608 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.958565    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-data\") pod \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") "
Nov 03 00:32:10.958717 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.958609    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-workload-socket" pod "2f8b0c71-4f60-47e3-9a22-7f25e85b690c" (UID: "2f8b0c71-4f60-47e3-9a22-7f25e85b690c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.958717 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.958632    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-envoy\") pod \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\" (UID: \"2f8b0c71-4f60-47e3-9a22-7f25e85b690c\") "
Nov 03 00:32:10.959195 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.958773    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-workload-certs" pod "2f8b0c71-4f60-47e3-9a22-7f25e85b690c" (UID: "2f8b0c71-4f60-47e3-9a22-7f25e85b690c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.959302 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.958816    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-credential-socket" pod "2f8b0c71-4f60-47e3-9a22-7f25e85b690c" (UID: "2f8b0c71-4f60-47e3-9a22-7f25e85b690c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.959302 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.958873    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-data" pod "2f8b0c71-4f60-47e3-9a22-7f25e85b690c" (UID: "2f8b0c71-4f60-47e3-9a22-7f25e85b690c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.959302 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.958928    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istiod-ca-cert" pod "2f8b0c71-4f60-47e3-9a22-7f25e85b690c" (UID: "2f8b0c71-4f60-47e3-9a22-7f25e85b690c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:10.959503 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.959369    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.959503 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.959391    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.959503 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.959406    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.959503 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.959419    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.959503 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.959433    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.960822 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.960800    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-podinfo" pod "2f8b0c71-4f60-47e3-9a22-7f25e85b690c" (UID: "2f8b0c71-4f60-47e3-9a22-7f25e85b690c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:32:10.960952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.960855    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-kube-api-access-tx9f9" pod "2f8b0c71-4f60-47e3-9a22-7f25e85b690c" (UID: "2f8b0c71-4f60-47e3-9a22-7f25e85b690c"). InnerVolumeSpecName "kube-api-access-tx9f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:10.961015 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.960950    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-token" pod "2f8b0c71-4f60-47e3-9a22-7f25e85b690c" (UID: "2f8b0c71-4f60-47e3-9a22-7f25e85b690c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:10.961420 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:10.961405    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-envoy" pod "2f8b0c71-4f60-47e3-9a22-7f25e85b690c" (UID: "2f8b0c71-4f60-47e3-9a22-7f25e85b690c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:11.060733 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:11.060656    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:11.060733 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:11.060687    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:11.060733 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:11.060697    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx9f9\" (UniqueName: \"kubernetes.io/projected/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-kube-api-access-tx9f9\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:11.060733 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:11.060706    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8b0c71-4f60-47e3-9a22-7f25e85b690c-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:11.246296 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:11.246261    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0528abca-dac8-491c-b6ab-d183adf6bf8a" path="/var/lib/kubelet/pods/0528abca-dac8-491c-b6ab-d183adf6bf8a/volumes"
Nov 03 00:32:11.246919 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:11.246885    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72825023-1f03-47a3-a213-4cf1656e108a" path="/var/lib/kubelet/pods/72825023-1f03-47a3-a213-4cf1656e108a/volumes"
Nov 03 00:32:11.357374 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:11.357347    5012 generic.go:356] "Generic (PLEG): container finished" podID="2f8b0c71-4f60-47e3-9a22-7f25e85b690c" containerID="ca6073d57aee0e6959f3d0ac717d95d3dac931794354b30bde191282a161ccda" exitCode=0
Nov 03 00:32:11.357495 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:11.357397    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"
Nov 03 00:32:11.357495 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:11.357435    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd" event={"ID":"2f8b0c71-4f60-47e3-9a22-7f25e85b690c","Type":"ContainerDied","Data":"ca6073d57aee0e6959f3d0ac717d95d3dac931794354b30bde191282a161ccda"}
Nov 03 00:32:11.357495 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:11.357491    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd" event={"ID":"2f8b0c71-4f60-47e3-9a22-7f25e85b690c","Type":"ContainerDied","Data":"1b1b8719974466e0a65f538798bfc3f1c6f31bfd24080c1c0e97bb21072c5bc0"}
Nov 03 00:32:11.357655 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:11.357518    5012 scope.go:122] "RemoveContainer" containerID="ca6073d57aee0e6959f3d0ac717d95d3dac931794354b30bde191282a161ccda"
Nov 03 00:32:11.361620 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:11.361596    5012 generic.go:356] "Generic (PLEG): container finished" podID="8760ca6d-230b-4d95-ae34-d9384b441607" containerID="193de80f706d976677f9e6fe90c33184d014cb5dbe1f3a6691ffa63646e3da05" exitCode=0
Nov 03 00:32:11.361729 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:11.361633    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"
Nov 03 00:32:11.362157 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:11.362121    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd" event={"ID":"8760ca6d-230b-4d95-ae34-d9384b441607","Type":"ContainerDied","Data":"193de80f706d976677f9e6fe90c33184d014cb5dbe1f3a6691ffa63646e3da05"}
Nov 03 00:32:11.362307 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:11.362286    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd" event={"ID":"8760ca6d-230b-4d95-ae34-d9384b441607","Type":"ContainerDied","Data":"ae14250ddc1a20ec61cf9413022fa7506207fd5e61ac18937bb52b642697351d"}
Nov 03 00:32:11.380695 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:11.380664    5012 scope.go:122] "RemoveContainer" containerID="ca6073d57aee0e6959f3d0ac717d95d3dac931794354b30bde191282a161ccda"
Nov 03 00:32:11.381083 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:32:11.381059    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca6073d57aee0e6959f3d0ac717d95d3dac931794354b30bde191282a161ccda\": container with ID starting with ca6073d57aee0e6959f3d0ac717d95d3dac931794354b30bde191282a161ccda not found: ID does not exist" containerID="ca6073d57aee0e6959f3d0ac717d95d3dac931794354b30bde191282a161ccda"
Nov 03 00:32:11.381155 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:11.381099    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca6073d57aee0e6959f3d0ac717d95d3dac931794354b30bde191282a161ccda"} err="failed to get container status \"ca6073d57aee0e6959f3d0ac717d95d3dac931794354b30bde191282a161ccda\": rpc error: code = NotFound desc = could not find container \"ca6073d57aee0e6959f3d0ac717d95d3dac931794354b30bde191282a161ccda\": container with ID starting with ca6073d57aee0e6959f3d0ac717d95d3dac931794354b30bde191282a161ccda not found: ID does not exist"
Nov 03 00:32:11.381155 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:11.381123    5012 scope.go:122] "RemoveContainer" containerID="193de80f706d976677f9e6fe90c33184d014cb5dbe1f3a6691ffa63646e3da05"
Nov 03 00:32:11.390958 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:11.390938    5012 scope.go:122] "RemoveContainer" containerID="193de80f706d976677f9e6fe90c33184d014cb5dbe1f3a6691ffa63646e3da05"
Nov 03 00:32:11.391216 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:32:11.391192    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"193de80f706d976677f9e6fe90c33184d014cb5dbe1f3a6691ffa63646e3da05\": container with ID starting with 193de80f706d976677f9e6fe90c33184d014cb5dbe1f3a6691ffa63646e3da05 not found: ID does not exist" containerID="193de80f706d976677f9e6fe90c33184d014cb5dbe1f3a6691ffa63646e3da05"
Nov 03 00:32:11.391275 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:11.391217    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"193de80f706d976677f9e6fe90c33184d014cb5dbe1f3a6691ffa63646e3da05"} err="failed to get container status \"193de80f706d976677f9e6fe90c33184d014cb5dbe1f3a6691ffa63646e3da05\": rpc error: code = NotFound desc = could not find container \"193de80f706d976677f9e6fe90c33184d014cb5dbe1f3a6691ffa63646e3da05\": container with ID starting with 193de80f706d976677f9e6fe90c33184d014cb5dbe1f3a6691ffa63646e3da05 not found: ID does not exist"
Nov 03 00:32:11.405676 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:11.405650    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"]
Nov 03 00:32:11.415539 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:11.415516    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-ggcmd"]
Nov 03 00:32:11.438782 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:11.438748    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"]
Nov 03 00:32:11.444617 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:11.444591    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8dgpd"]
Nov 03 00:32:13.245318 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:13.245284    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f8b0c71-4f60-47e3-9a22-7f25e85b690c" path="/var/lib/kubelet/pods/2f8b0c71-4f60-47e3-9a22-7f25e85b690c/volumes"
Nov 03 00:32:13.245767 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:13.245753    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8760ca6d-230b-4d95-ae34-d9384b441607" path="/var/lib/kubelet/pods/8760ca6d-230b-4d95-ae34-d9384b441607/volumes"
Nov 03 00:32:13.894318 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:13.894253    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"]
Nov 03 00:32:13.894762 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:13.894726    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg" podUID="90b16caf-d83f-4c7b-a1fc-bdcb7d061653" containerName="istio-proxy" containerID="cri-o://d49bd4b396cb8648aea74c1b4a1a1c8b5dc65af4f32f9e7c5aa22d3600ec3d75" gracePeriod=30
Nov 03 00:32:14.865317 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:14.865286    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl"]
Nov 03 00:32:14.913492 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:14.913462    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc"]
Nov 03 00:32:17.365561 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:17.365527    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"]
Nov 03 00:32:19.183175 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.183082    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:32:19.292514 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.292479    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-podinfo\") pod \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") "
Nov 03 00:32:19.292698 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.292530    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-data\") pod \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") "
Nov 03 00:32:19.292698 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.292558    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-envoy\") pod \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") "
Nov 03 00:32:19.292698 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.292593    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-token\" (UniqueName: \"kubernetes.io/projected/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-token\") pod \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") "
Nov 03 00:32:19.292698 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.292660    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istiod-ca-cert\") pod \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") "
Nov 03 00:32:19.292933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.292722    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-workload-certs\") pod \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") "
Nov 03 00:32:19.292933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.292764    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-workload-socket\") pod \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") "
Nov 03 00:32:19.293059 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.292948    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-data" pod "90b16caf-d83f-4c7b-a1fc-bdcb7d061653" (UID: "90b16caf-d83f-4c7b-a1fc-bdcb7d061653"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:19.293059 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.292982    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-kube-api-access-7r85z\" (UniqueName: \"kubernetes.io/projected/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-kube-api-access-7r85z\") pod \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") "
Nov 03 00:32:19.293059 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.293035    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-credential-socket\") pod \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\" (UID: \"90b16caf-d83f-4c7b-a1fc-bdcb7d061653\") "
Nov 03 00:32:19.293219 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.293118    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-workload-certs" pod "90b16caf-d83f-4c7b-a1fc-bdcb7d061653" (UID: "90b16caf-d83f-4c7b-a1fc-bdcb7d061653"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:19.293279 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.293233    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-workload-socket" pod "90b16caf-d83f-4c7b-a1fc-bdcb7d061653" (UID: "90b16caf-d83f-4c7b-a1fc-bdcb7d061653"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:19.293396 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.293377    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-credential-socket" pod "90b16caf-d83f-4c7b-a1fc-bdcb7d061653" (UID: "90b16caf-d83f-4c7b-a1fc-bdcb7d061653"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:19.293466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.293441    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istiod-ca-cert" pod "90b16caf-d83f-4c7b-a1fc-bdcb7d061653" (UID: "90b16caf-d83f-4c7b-a1fc-bdcb7d061653"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:19.293839 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.293815    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:19.293993 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.293841    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:19.293993 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.293856    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:19.293993 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.293870    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:19.293993 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.293884    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:19.295290 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.295266    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-podinfo" pod "90b16caf-d83f-4c7b-a1fc-bdcb7d061653" (UID: "90b16caf-d83f-4c7b-a1fc-bdcb7d061653"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:32:19.295380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.295264    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-envoy" pod "90b16caf-d83f-4c7b-a1fc-bdcb7d061653" (UID: "90b16caf-d83f-4c7b-a1fc-bdcb7d061653"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:19.295498 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.295483    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-token" pod "90b16caf-d83f-4c7b-a1fc-bdcb7d061653" (UID: "90b16caf-d83f-4c7b-a1fc-bdcb7d061653"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:19.295950 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.295903    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-kube-api-access-7r85z" pod "90b16caf-d83f-4c7b-a1fc-bdcb7d061653" (UID: "90b16caf-d83f-4c7b-a1fc-bdcb7d061653"). InnerVolumeSpecName "kube-api-access-7r85z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:19.395405 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.395376    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r85z\" (UniqueName: \"kubernetes.io/projected/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-kube-api-access-7r85z\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:19.395405 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.395404    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:19.395563 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.395413    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:19.395563 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.395422    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b16caf-d83f-4c7b-a1fc-bdcb7d061653-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:19.429345 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.429317    5012 generic.go:356] "Generic (PLEG): container finished" podID="90b16caf-d83f-4c7b-a1fc-bdcb7d061653" containerID="d49bd4b396cb8648aea74c1b4a1a1c8b5dc65af4f32f9e7c5aa22d3600ec3d75" exitCode=0
Nov 03 00:32:19.429455 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.429374    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"
Nov 03 00:32:19.429455 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.429379    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg" event={"ID":"90b16caf-d83f-4c7b-a1fc-bdcb7d061653","Type":"ContainerDied","Data":"d49bd4b396cb8648aea74c1b4a1a1c8b5dc65af4f32f9e7c5aa22d3600ec3d75"}
Nov 03 00:32:19.429455 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.429423    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg" event={"ID":"90b16caf-d83f-4c7b-a1fc-bdcb7d061653","Type":"ContainerDied","Data":"fdaaea1dfc462ef1d2559cc9e430b3dc8ed607b8d6246addbaef10f8cd167257"}
Nov 03 00:32:19.429455 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.429449    5012 scope.go:122] "RemoveContainer" containerID="d49bd4b396cb8648aea74c1b4a1a1c8b5dc65af4f32f9e7c5aa22d3600ec3d75"
Nov 03 00:32:19.443221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.443141    5012 scope.go:122] "RemoveContainer" containerID="fd9b8edd5fd21bf21e78c6917f58e7c47fe4713f1a392f53830351f7ca1c6a7e"
Nov 03 00:32:19.458885 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.458864    5012 scope.go:122] "RemoveContainer" containerID="d49bd4b396cb8648aea74c1b4a1a1c8b5dc65af4f32f9e7c5aa22d3600ec3d75"
Nov 03 00:32:19.459194 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:32:19.459171    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d49bd4b396cb8648aea74c1b4a1a1c8b5dc65af4f32f9e7c5aa22d3600ec3d75\": container with ID starting with d49bd4b396cb8648aea74c1b4a1a1c8b5dc65af4f32f9e7c5aa22d3600ec3d75 not found: ID does not exist" containerID="d49bd4b396cb8648aea74c1b4a1a1c8b5dc65af4f32f9e7c5aa22d3600ec3d75"
Nov 03 00:32:19.459245 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.459204    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d49bd4b396cb8648aea74c1b4a1a1c8b5dc65af4f32f9e7c5aa22d3600ec3d75"} err="failed to get container status \"d49bd4b396cb8648aea74c1b4a1a1c8b5dc65af4f32f9e7c5aa22d3600ec3d75\": rpc error: code = NotFound desc = could not find container \"d49bd4b396cb8648aea74c1b4a1a1c8b5dc65af4f32f9e7c5aa22d3600ec3d75\": container with ID starting with d49bd4b396cb8648aea74c1b4a1a1c8b5dc65af4f32f9e7c5aa22d3600ec3d75 not found: ID does not exist"
Nov 03 00:32:19.459245 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.459226    5012 scope.go:122] "RemoveContainer" containerID="fd9b8edd5fd21bf21e78c6917f58e7c47fe4713f1a392f53830351f7ca1c6a7e"
Nov 03 00:32:19.459503 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:32:19.459485    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd9b8edd5fd21bf21e78c6917f58e7c47fe4713f1a392f53830351f7ca1c6a7e\": container with ID starting with fd9b8edd5fd21bf21e78c6917f58e7c47fe4713f1a392f53830351f7ca1c6a7e not found: ID does not exist" containerID="fd9b8edd5fd21bf21e78c6917f58e7c47fe4713f1a392f53830351f7ca1c6a7e"
Nov 03 00:32:19.459549 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.459511    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9b8edd5fd21bf21e78c6917f58e7c47fe4713f1a392f53830351f7ca1c6a7e"} err="failed to get container status \"fd9b8edd5fd21bf21e78c6917f58e7c47fe4713f1a392f53830351f7ca1c6a7e\": rpc error: code = NotFound desc = could not find container \"fd9b8edd5fd21bf21e78c6917f58e7c47fe4713f1a392f53830351f7ca1c6a7e\": container with ID starting with fd9b8edd5fd21bf21e78c6917f58e7c47fe4713f1a392f53830351f7ca1c6a7e not found: ID does not exist"
Nov 03 00:32:19.478798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.478778    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"]
Nov 03 00:32:19.482694 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.482674    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ea275f-openshift-default-68b6755bd9-t9tdg"]
Nov 03 00:32:19.865733 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.865695    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl" podUID="83f374c7-f0aa-46c2-beca-544aaf65a7b7" containerName="payload-processing" containerID="cri-o://d7b85ef8d446901729e625e36e030019bae1b989f2eed0dcd7cd744b9352f7fd" gracePeriod=25
Nov 03 00:32:19.923545 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.914882    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc" podUID="1f594be4-6f2b-48f1-9fef-34a19645c6dc" containerName="payload-pre-processing" containerID="cri-o://2cf89ab81baf4c5d2a55ce26bd7c05892309f068eb9c303bfbb0fae800d78710" gracePeriod=25
Nov 03 00:32:19.939437 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.939115    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"]
Nov 03 00:32:19.939437 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:19.939347    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz" podUID="5e8e36a8-3445-41ab-a889-c529c3e707c9" containerName="istio-proxy" containerID="cri-o://24376c343bb977be3c828b2c83525432a46508f6e61eed1c86441a12f6b1e155" gracePeriod=30
Nov 03 00:32:20.346788 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.346761    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc"
Nov 03 00:32:20.388063 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.388035    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl"
Nov 03 00:32:20.423522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.423492    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83f374c7-f0aa-46c2-beca-544aaf65a7b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83f374c7-f0aa-46c2-beca-544aaf65a7b7-plugins-config-volume\") pod \"83f374c7-f0aa-46c2-beca-544aaf65a7b7\" (UID: \"83f374c7-f0aa-46c2-beca-544aaf65a7b7\") "
Nov 03 00:32:20.423709 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.423540    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f594be4-6f2b-48f1-9fef-34a19645c6dc-kube-api-access-5sxbn\" (UniqueName: \"kubernetes.io/projected/1f594be4-6f2b-48f1-9fef-34a19645c6dc-kube-api-access-5sxbn\") pod \"1f594be4-6f2b-48f1-9fef-34a19645c6dc\" (UID: \"1f594be4-6f2b-48f1-9fef-34a19645c6dc\") "
Nov 03 00:32:20.423709 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.423656    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83f374c7-f0aa-46c2-beca-544aaf65a7b7-kube-api-access-gbm6g\" (UniqueName: \"kubernetes.io/projected/83f374c7-f0aa-46c2-beca-544aaf65a7b7-kube-api-access-gbm6g\") pod \"83f374c7-f0aa-46c2-beca-544aaf65a7b7\" (UID: \"83f374c7-f0aa-46c2-beca-544aaf65a7b7\") "
Nov 03 00:32:20.424954 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.424205    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f594be4-6f2b-48f1-9fef-34a19645c6dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f594be4-6f2b-48f1-9fef-34a19645c6dc-plugins-config-volume\") pod \"1f594be4-6f2b-48f1-9fef-34a19645c6dc\" (UID: \"1f594be4-6f2b-48f1-9fef-34a19645c6dc\") "
Nov 03 00:32:20.425128 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.425104    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f594be4-6f2b-48f1-9fef-34a19645c6dc-plugins-config-volume" pod "1f594be4-6f2b-48f1-9fef-34a19645c6dc" (UID: "1f594be4-6f2b-48f1-9fef-34a19645c6dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:20.425517 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.425496    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f374c7-f0aa-46c2-beca-544aaf65a7b7-plugins-config-volume" pod "83f374c7-f0aa-46c2-beca-544aaf65a7b7" (UID: "83f374c7-f0aa-46c2-beca-544aaf65a7b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:20.428151 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.428121    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f374c7-f0aa-46c2-beca-544aaf65a7b7-kube-api-access-gbm6g" pod "83f374c7-f0aa-46c2-beca-544aaf65a7b7" (UID: "83f374c7-f0aa-46c2-beca-544aaf65a7b7"). InnerVolumeSpecName "kube-api-access-gbm6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:20.429756 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.429044    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f594be4-6f2b-48f1-9fef-34a19645c6dc-kube-api-access-5sxbn" pod "1f594be4-6f2b-48f1-9fef-34a19645c6dc" (UID: "1f594be4-6f2b-48f1-9fef-34a19645c6dc"). InnerVolumeSpecName "kube-api-access-5sxbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:20.443031 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.442842    5012 generic.go:356] "Generic (PLEG): container finished" podID="1f594be4-6f2b-48f1-9fef-34a19645c6dc" containerID="2cf89ab81baf4c5d2a55ce26bd7c05892309f068eb9c303bfbb0fae800d78710" exitCode=0
Nov 03 00:32:20.443031 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.442892    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc" event={"ID":"1f594be4-6f2b-48f1-9fef-34a19645c6dc","Type":"ContainerDied","Data":"2cf89ab81baf4c5d2a55ce26bd7c05892309f068eb9c303bfbb0fae800d78710"}
Nov 03 00:32:20.443031 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.443002    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc" event={"ID":"1f594be4-6f2b-48f1-9fef-34a19645c6dc","Type":"ContainerDied","Data":"806b0cb1355c03752f5462a9043e10adcd047fc7ef054d1bb1d9a5f2e5a466f2"}
Nov 03 00:32:20.443321 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.443071    5012 scope.go:122] "RemoveContainer" containerID="2cf89ab81baf4c5d2a55ce26bd7c05892309f068eb9c303bfbb0fae800d78710"
Nov 03 00:32:20.444004 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.443972    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc"
Nov 03 00:32:20.450528 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.450501    5012 generic.go:356] "Generic (PLEG): container finished" podID="83f374c7-f0aa-46c2-beca-544aaf65a7b7" containerID="d7b85ef8d446901729e625e36e030019bae1b989f2eed0dcd7cd744b9352f7fd" exitCode=0
Nov 03 00:32:20.450644 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.450552    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl"
Nov 03 00:32:20.450644 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.450579    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl" event={"ID":"83f374c7-f0aa-46c2-beca-544aaf65a7b7","Type":"ContainerDied","Data":"d7b85ef8d446901729e625e36e030019bae1b989f2eed0dcd7cd744b9352f7fd"}
Nov 03 00:32:20.450644 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.450621    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl" event={"ID":"83f374c7-f0aa-46c2-beca-544aaf65a7b7","Type":"ContainerDied","Data":"d77d25e8fa0b2e8e08e975eab127f938193cd41c35fe4db89a6730af07caf8b7"}
Nov 03 00:32:20.463132 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.463110    5012 scope.go:122] "RemoveContainer" containerID="2cf89ab81baf4c5d2a55ce26bd7c05892309f068eb9c303bfbb0fae800d78710"
Nov 03 00:32:20.463465 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:32:20.463444    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cf89ab81baf4c5d2a55ce26bd7c05892309f068eb9c303bfbb0fae800d78710\": container with ID starting with 2cf89ab81baf4c5d2a55ce26bd7c05892309f068eb9c303bfbb0fae800d78710 not found: ID does not exist" containerID="2cf89ab81baf4c5d2a55ce26bd7c05892309f068eb9c303bfbb0fae800d78710"
Nov 03 00:32:20.463545 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.463478    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cf89ab81baf4c5d2a55ce26bd7c05892309f068eb9c303bfbb0fae800d78710"} err="failed to get container status \"2cf89ab81baf4c5d2a55ce26bd7c05892309f068eb9c303bfbb0fae800d78710\": rpc error: code = NotFound desc = could not find container \"2cf89ab81baf4c5d2a55ce26bd7c05892309f068eb9c303bfbb0fae800d78710\": container with ID starting with 2cf89ab81baf4c5d2a55ce26bd7c05892309f068eb9c303bfbb0fae800d78710 not found: ID does not exist"
Nov 03 00:32:20.463545 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.463504    5012 scope.go:122] "RemoveContainer" containerID="d7b85ef8d446901729e625e36e030019bae1b989f2eed0dcd7cd744b9352f7fd"
Nov 03 00:32:20.474726 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.474704    5012 scope.go:122] "RemoveContainer" containerID="d7b85ef8d446901729e625e36e030019bae1b989f2eed0dcd7cd744b9352f7fd"
Nov 03 00:32:20.475057 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:32:20.475032    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b85ef8d446901729e625e36e030019bae1b989f2eed0dcd7cd744b9352f7fd\": container with ID starting with d7b85ef8d446901729e625e36e030019bae1b989f2eed0dcd7cd744b9352f7fd not found: ID does not exist" containerID="d7b85ef8d446901729e625e36e030019bae1b989f2eed0dcd7cd744b9352f7fd"
Nov 03 00:32:20.475145 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.475063    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b85ef8d446901729e625e36e030019bae1b989f2eed0dcd7cd744b9352f7fd"} err="failed to get container status \"d7b85ef8d446901729e625e36e030019bae1b989f2eed0dcd7cd744b9352f7fd\": rpc error: code = NotFound desc = could not find container \"d7b85ef8d446901729e625e36e030019bae1b989f2eed0dcd7cd744b9352f7fd\": container with ID starting with d7b85ef8d446901729e625e36e030019bae1b989f2eed0dcd7cd744b9352f7fd not found: ID does not exist"
Nov 03 00:32:20.492121 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.492096    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc"]
Nov 03 00:32:20.495953 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.495929    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2df502-684f5d97c5-tqkfc"]
Nov 03 00:32:20.526493 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.526465    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f594be4-6f2b-48f1-9fef-34a19645c6dc-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:20.526493 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.526488    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83f374c7-f0aa-46c2-beca-544aaf65a7b7-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:20.526640 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.526498    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sxbn\" (UniqueName: \"kubernetes.io/projected/1f594be4-6f2b-48f1-9fef-34a19645c6dc-kube-api-access-5sxbn\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:20.526640 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.526508    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbm6g\" (UniqueName: \"kubernetes.io/projected/83f374c7-f0aa-46c2-beca-544aaf65a7b7-kube-api-access-gbm6g\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:20.536854 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.536827    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl"]
Nov 03 00:32:20.540850 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:20.540827    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2df502-6597947bdf-7whbl"]
Nov 03 00:32:21.246251 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:21.246176    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f594be4-6f2b-48f1-9fef-34a19645c6dc" path="/var/lib/kubelet/pods/1f594be4-6f2b-48f1-9fef-34a19645c6dc/volumes"
Nov 03 00:32:21.246591 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:21.246577    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83f374c7-f0aa-46c2-beca-544aaf65a7b7" path="/var/lib/kubelet/pods/83f374c7-f0aa-46c2-beca-544aaf65a7b7/volumes"
Nov 03 00:32:21.247018 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:21.247005    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90b16caf-d83f-4c7b-a1fc-bdcb7d061653" path="/var/lib/kubelet/pods/90b16caf-d83f-4c7b-a1fc-bdcb7d061653/volumes"
Nov 03 00:32:25.228794 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.228769    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:32:25.303453 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.303422    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-token\" (UniqueName: \"kubernetes.io/projected/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-token\") pod \"5e8e36a8-3445-41ab-a889-c529c3e707c9\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") "
Nov 03 00:32:25.303615 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.303484    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-data\") pod \"5e8e36a8-3445-41ab-a889-c529c3e707c9\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") "
Nov 03 00:32:25.303699 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.303611    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-workload-socket\") pod \"5e8e36a8-3445-41ab-a889-c529c3e707c9\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") "
Nov 03 00:32:25.303759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.303703    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-envoy\") pod \"5e8e36a8-3445-41ab-a889-c529c3e707c9\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") "
Nov 03 00:32:25.303759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.303736    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-credential-socket\") pod \"5e8e36a8-3445-41ab-a889-c529c3e707c9\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") "
Nov 03 00:32:25.303868 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.303814    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e8e36a8-3445-41ab-a889-c529c3e707c9-kube-api-access-k58h9\" (UniqueName: \"kubernetes.io/projected/5e8e36a8-3445-41ab-a889-c529c3e707c9-kube-api-access-k58h9\") pod \"5e8e36a8-3445-41ab-a889-c529c3e707c9\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") "
Nov 03 00:32:25.303868 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.303844    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e8e36a8-3445-41ab-a889-c529c3e707c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8e36a8-3445-41ab-a889-c529c3e707c9-istiod-ca-cert\") pod \"5e8e36a8-3445-41ab-a889-c529c3e707c9\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") "
Nov 03 00:32:25.304075 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.303866    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-data" pod "5e8e36a8-3445-41ab-a889-c529c3e707c9" (UID: "5e8e36a8-3445-41ab-a889-c529c3e707c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:25.304075 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.303952    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-podinfo\") pod \"5e8e36a8-3445-41ab-a889-c529c3e707c9\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") "
Nov 03 00:32:25.304075 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.304035    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-workload-certs\") pod \"5e8e36a8-3445-41ab-a889-c529c3e707c9\" (UID: \"5e8e36a8-3445-41ab-a889-c529c3e707c9\") "
Nov 03 00:32:25.305621 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.304612    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e8e36a8-3445-41ab-a889-c529c3e707c9-istiod-ca-cert" pod "5e8e36a8-3445-41ab-a889-c529c3e707c9" (UID: "5e8e36a8-3445-41ab-a889-c529c3e707c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:25.305621 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.304714    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:25.305621 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.304737    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8e36a8-3445-41ab-a889-c529c3e707c9-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:25.305621 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.305158    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-credential-socket" pod "5e8e36a8-3445-41ab-a889-c529c3e707c9" (UID: "5e8e36a8-3445-41ab-a889-c529c3e707c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:25.305621 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.305540    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-workload-socket" pod "5e8e36a8-3445-41ab-a889-c529c3e707c9" (UID: "5e8e36a8-3445-41ab-a889-c529c3e707c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:25.305621 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.305596    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-workload-certs" pod "5e8e36a8-3445-41ab-a889-c529c3e707c9" (UID: "5e8e36a8-3445-41ab-a889-c529c3e707c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:25.306432 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.306404    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-envoy" pod "5e8e36a8-3445-41ab-a889-c529c3e707c9" (UID: "5e8e36a8-3445-41ab-a889-c529c3e707c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:25.307264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.307232    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-token" pod "5e8e36a8-3445-41ab-a889-c529c3e707c9" (UID: "5e8e36a8-3445-41ab-a889-c529c3e707c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:25.307378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.307355    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8e36a8-3445-41ab-a889-c529c3e707c9-kube-api-access-k58h9" pod "5e8e36a8-3445-41ab-a889-c529c3e707c9" (UID: "5e8e36a8-3445-41ab-a889-c529c3e707c9"). InnerVolumeSpecName "kube-api-access-k58h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:25.308164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.308140    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-podinfo" pod "5e8e36a8-3445-41ab-a889-c529c3e707c9" (UID: "5e8e36a8-3445-41ab-a889-c529c3e707c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:32:25.406482 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.406456    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:25.406482 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.406481    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:25.406652 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.406491    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:25.406652 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.406499    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k58h9\" (UniqueName: \"kubernetes.io/projected/5e8e36a8-3445-41ab-a889-c529c3e707c9-kube-api-access-k58h9\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:25.406652 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.406510    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:25.406652 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.406522    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8e36a8-3445-41ab-a889-c529c3e707c9-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:25.406652 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.406530    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8e36a8-3445-41ab-a889-c529c3e707c9-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:25.498093 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.497994    5012 generic.go:356] "Generic (PLEG): container finished" podID="5e8e36a8-3445-41ab-a889-c529c3e707c9" containerID="24376c343bb977be3c828b2c83525432a46508f6e61eed1c86441a12f6b1e155" exitCode=0
Nov 03 00:32:25.498093 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.498056    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"
Nov 03 00:32:25.498280 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.498090    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz" event={"ID":"5e8e36a8-3445-41ab-a889-c529c3e707c9","Type":"ContainerDied","Data":"24376c343bb977be3c828b2c83525432a46508f6e61eed1c86441a12f6b1e155"}
Nov 03 00:32:25.498280 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.498146    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz" event={"ID":"5e8e36a8-3445-41ab-a889-c529c3e707c9","Type":"ContainerDied","Data":"0d503760a94923602c5951f41b1e0f9ff18ce21a32609c44bf7712c3493a2131"}
Nov 03 00:32:25.498280 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.498167    5012 scope.go:122] "RemoveContainer" containerID="24376c343bb977be3c828b2c83525432a46508f6e61eed1c86441a12f6b1e155"
Nov 03 00:32:25.512271 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.512253    5012 scope.go:122] "RemoveContainer" containerID="cd96a43eed3105271ace6242ac2fc21431a985189829b6a578bf4b6d381089ab"
Nov 03 00:32:25.522101 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.522081    5012 scope.go:122] "RemoveContainer" containerID="24376c343bb977be3c828b2c83525432a46508f6e61eed1c86441a12f6b1e155"
Nov 03 00:32:25.522400 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:32:25.522383    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24376c343bb977be3c828b2c83525432a46508f6e61eed1c86441a12f6b1e155\": container with ID starting with 24376c343bb977be3c828b2c83525432a46508f6e61eed1c86441a12f6b1e155 not found: ID does not exist" containerID="24376c343bb977be3c828b2c83525432a46508f6e61eed1c86441a12f6b1e155"
Nov 03 00:32:25.522476 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.522408    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24376c343bb977be3c828b2c83525432a46508f6e61eed1c86441a12f6b1e155"} err="failed to get container status \"24376c343bb977be3c828b2c83525432a46508f6e61eed1c86441a12f6b1e155\": rpc error: code = NotFound desc = could not find container \"24376c343bb977be3c828b2c83525432a46508f6e61eed1c86441a12f6b1e155\": container with ID starting with 24376c343bb977be3c828b2c83525432a46508f6e61eed1c86441a12f6b1e155 not found: ID does not exist"
Nov 03 00:32:25.522476 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.522431    5012 scope.go:122] "RemoveContainer" containerID="cd96a43eed3105271ace6242ac2fc21431a985189829b6a578bf4b6d381089ab"
Nov 03 00:32:25.522694 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:32:25.522677    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd96a43eed3105271ace6242ac2fc21431a985189829b6a578bf4b6d381089ab\": container with ID starting with cd96a43eed3105271ace6242ac2fc21431a985189829b6a578bf4b6d381089ab not found: ID does not exist" containerID="cd96a43eed3105271ace6242ac2fc21431a985189829b6a578bf4b6d381089ab"
Nov 03 00:32:25.522737 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.522701    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd96a43eed3105271ace6242ac2fc21431a985189829b6a578bf4b6d381089ab"} err="failed to get container status \"cd96a43eed3105271ace6242ac2fc21431a985189829b6a578bf4b6d381089ab\": rpc error: code = NotFound desc = could not find container \"cd96a43eed3105271ace6242ac2fc21431a985189829b6a578bf4b6d381089ab\": container with ID starting with cd96a43eed3105271ace6242ac2fc21431a985189829b6a578bf4b6d381089ab not found: ID does not exist"
Nov 03 00:32:25.546067 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.546033    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"]
Nov 03 00:32:25.552028 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:25.552006    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-846492-openshift-default-65c467f785-q9kpz"]
Nov 03 00:32:27.254149 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:27.254114    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e8e36a8-3445-41ab-a889-c529c3e707c9" path="/var/lib/kubelet/pods/5e8e36a8-3445-41ab-a889-c529c3e707c9/volumes"
Nov 03 00:32:30.712004 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:30.711961    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g"]
Nov 03 00:32:30.712422 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:30.712197    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g" podUID="b2570d35-c9ee-4069-85c2-361cba7c27fb" containerName="payload-processing" containerID="cri-o://d6c7292bb52a34cad1f20904fb7763db99739a7eef591c1c31f807dd3a976b0c" gracePeriod=30
Nov 03 00:32:30.746553 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:30.744025    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg"]
Nov 03 00:32:30.746553 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:30.744315    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg" podUID="9ac19255-340c-4891-85d4-1b895e2d3641" containerName="payload-pre-processing" containerID="cri-o://343f4a9bb285af96e3e41b3b2a8f4d77688ece2ca0eb55081163e8da3e3ecd34" gracePeriod=30
Nov 03 00:32:31.152124 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.152091    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g"
Nov 03 00:32:31.198514 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.198481    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg"
Nov 03 00:32:31.238586 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.238553    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2570d35-c9ee-4069-85c2-361cba7c27fb-kube-api-access-5689r\" (UniqueName: \"kubernetes.io/projected/b2570d35-c9ee-4069-85c2-361cba7c27fb-kube-api-access-5689r\") pod \"b2570d35-c9ee-4069-85c2-361cba7c27fb\" (UID: \"b2570d35-c9ee-4069-85c2-361cba7c27fb\") "
Nov 03 00:32:31.239385 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.239361    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2570d35-c9ee-4069-85c2-361cba7c27fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2570d35-c9ee-4069-85c2-361cba7c27fb-plugins-config-volume\") pod \"b2570d35-c9ee-4069-85c2-361cba7c27fb\" (UID: \"b2570d35-c9ee-4069-85c2-361cba7c27fb\") "
Nov 03 00:32:31.239790 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.239771    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ac19255-340c-4891-85d4-1b895e2d3641-kube-api-access-85456\" (UniqueName: \"kubernetes.io/projected/9ac19255-340c-4891-85d4-1b895e2d3641-kube-api-access-85456\") pod \"9ac19255-340c-4891-85d4-1b895e2d3641\" (UID: \"9ac19255-340c-4891-85d4-1b895e2d3641\") "
Nov 03 00:32:31.240031 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.239872    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2570d35-c9ee-4069-85c2-361cba7c27fb-plugins-config-volume" pod "b2570d35-c9ee-4069-85c2-361cba7c27fb" (UID: "b2570d35-c9ee-4069-85c2-361cba7c27fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:31.240261 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.240238    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ac19255-340c-4891-85d4-1b895e2d3641-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac19255-340c-4891-85d4-1b895e2d3641-plugins-config-volume\") pod \"9ac19255-340c-4891-85d4-1b895e2d3641\" (UID: \"9ac19255-340c-4891-85d4-1b895e2d3641\") "
Nov 03 00:32:31.240785 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.240747    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac19255-340c-4891-85d4-1b895e2d3641-plugins-config-volume" pod "9ac19255-340c-4891-85d4-1b895e2d3641" (UID: "9ac19255-340c-4891-85d4-1b895e2d3641"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:31.242238 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.242213    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2570d35-c9ee-4069-85c2-361cba7c27fb-kube-api-access-5689r" pod "b2570d35-c9ee-4069-85c2-361cba7c27fb" (UID: "b2570d35-c9ee-4069-85c2-361cba7c27fb"). InnerVolumeSpecName "kube-api-access-5689r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:31.242486 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.242468    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac19255-340c-4891-85d4-1b895e2d3641-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:31.242588 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.242491    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5689r\" (UniqueName: \"kubernetes.io/projected/b2570d35-c9ee-4069-85c2-361cba7c27fb-kube-api-access-5689r\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:31.242588 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.242506    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2570d35-c9ee-4069-85c2-361cba7c27fb-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:31.248160 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.248110    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac19255-340c-4891-85d4-1b895e2d3641-kube-api-access-85456" pod "9ac19255-340c-4891-85d4-1b895e2d3641" (UID: "9ac19255-340c-4891-85d4-1b895e2d3641"). InnerVolumeSpecName "kube-api-access-85456". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:31.343890 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.343853    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85456\" (UniqueName: \"kubernetes.io/projected/9ac19255-340c-4891-85d4-1b895e2d3641-kube-api-access-85456\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:31.561634 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.561552    5012 generic.go:356] "Generic (PLEG): container finished" podID="b2570d35-c9ee-4069-85c2-361cba7c27fb" containerID="d6c7292bb52a34cad1f20904fb7763db99739a7eef591c1c31f807dd3a976b0c" exitCode=0
Nov 03 00:32:31.561634 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.561598    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g"
Nov 03 00:32:31.561830 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.561636    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g" event={"ID":"b2570d35-c9ee-4069-85c2-361cba7c27fb","Type":"ContainerDied","Data":"d6c7292bb52a34cad1f20904fb7763db99739a7eef591c1c31f807dd3a976b0c"}
Nov 03 00:32:31.561830 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.561689    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g" event={"ID":"b2570d35-c9ee-4069-85c2-361cba7c27fb","Type":"ContainerDied","Data":"a6f27e94ef171e80b06a23fbfeb2c0fcfac0c698e43c0b87615ff3c92f917fef"}
Nov 03 00:32:31.561830 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.561711    5012 scope.go:122] "RemoveContainer" containerID="d6c7292bb52a34cad1f20904fb7763db99739a7eef591c1c31f807dd3a976b0c"
Nov 03 00:32:31.563490 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.563424    5012 generic.go:356] "Generic (PLEG): container finished" podID="9ac19255-340c-4891-85d4-1b895e2d3641" containerID="343f4a9bb285af96e3e41b3b2a8f4d77688ece2ca0eb55081163e8da3e3ecd34" exitCode=0
Nov 03 00:32:31.563490 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.563452    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg" event={"ID":"9ac19255-340c-4891-85d4-1b895e2d3641","Type":"ContainerDied","Data":"343f4a9bb285af96e3e41b3b2a8f4d77688ece2ca0eb55081163e8da3e3ecd34"}
Nov 03 00:32:31.563490 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.563468    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg"
Nov 03 00:32:31.563740 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.563502    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg" event={"ID":"9ac19255-340c-4891-85d4-1b895e2d3641","Type":"ContainerDied","Data":"fffd7d60ba3dc921efc3614c3c0b9306a950f6e18de3d3d3761db6ac4074d441"}
Nov 03 00:32:31.572768 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.572751    5012 scope.go:122] "RemoveContainer" containerID="d6c7292bb52a34cad1f20904fb7763db99739a7eef591c1c31f807dd3a976b0c"
Nov 03 00:32:31.573072 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:32:31.573054    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6c7292bb52a34cad1f20904fb7763db99739a7eef591c1c31f807dd3a976b0c\": container with ID starting with d6c7292bb52a34cad1f20904fb7763db99739a7eef591c1c31f807dd3a976b0c not found: ID does not exist" containerID="d6c7292bb52a34cad1f20904fb7763db99739a7eef591c1c31f807dd3a976b0c"
Nov 03 00:32:31.573126 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.573080    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c7292bb52a34cad1f20904fb7763db99739a7eef591c1c31f807dd3a976b0c"} err="failed to get container status \"d6c7292bb52a34cad1f20904fb7763db99739a7eef591c1c31f807dd3a976b0c\": rpc error: code = NotFound desc = could not find container \"d6c7292bb52a34cad1f20904fb7763db99739a7eef591c1c31f807dd3a976b0c\": container with ID starting with d6c7292bb52a34cad1f20904fb7763db99739a7eef591c1c31f807dd3a976b0c not found: ID does not exist"
Nov 03 00:32:31.573126 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.573096    5012 scope.go:122] "RemoveContainer" containerID="343f4a9bb285af96e3e41b3b2a8f4d77688ece2ca0eb55081163e8da3e3ecd34"
Nov 03 00:32:31.582413 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.582393    5012 scope.go:122] "RemoveContainer" containerID="343f4a9bb285af96e3e41b3b2a8f4d77688ece2ca0eb55081163e8da3e3ecd34"
Nov 03 00:32:31.582667 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:32:31.582650    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"343f4a9bb285af96e3e41b3b2a8f4d77688ece2ca0eb55081163e8da3e3ecd34\": container with ID starting with 343f4a9bb285af96e3e41b3b2a8f4d77688ece2ca0eb55081163e8da3e3ecd34 not found: ID does not exist" containerID="343f4a9bb285af96e3e41b3b2a8f4d77688ece2ca0eb55081163e8da3e3ecd34"
Nov 03 00:32:31.582733 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.582672    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343f4a9bb285af96e3e41b3b2a8f4d77688ece2ca0eb55081163e8da3e3ecd34"} err="failed to get container status \"343f4a9bb285af96e3e41b3b2a8f4d77688ece2ca0eb55081163e8da3e3ecd34\": rpc error: code = NotFound desc = could not find container \"343f4a9bb285af96e3e41b3b2a8f4d77688ece2ca0eb55081163e8da3e3ecd34\": container with ID starting with 343f4a9bb285af96e3e41b3b2a8f4d77688ece2ca0eb55081163e8da3e3ecd34 not found: ID does not exist"
Nov 03 00:32:31.609199 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.609168    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g"]
Nov 03 00:32:31.612832 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.612809    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-5d6b5884b-sdn5g"]
Nov 03 00:32:31.642026 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.641991    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg"]
Nov 03 00:32:31.645804 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:31.645776    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-57b469ccc5-bc7zg"]
Nov 03 00:32:32.387844 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:32.387797    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n" podUID="02a9089a-7dee-4192-b1c3-3ccc297c8d4f" containerName="main" containerID="cri-o://b7dff37044ce76116760ee47170d9a6dff6d02291fe8cd690a5129aa36f40a3a" gracePeriod=45
Nov 03 00:32:32.583636 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:32.583592    5012 generic.go:356] "Generic (PLEG): container finished" podID="02a9089a-7dee-4192-b1c3-3ccc297c8d4f" containerID="b7dff37044ce76116760ee47170d9a6dff6d02291fe8cd690a5129aa36f40a3a" exitCode=0
Nov 03 00:32:32.583849 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:32.583662    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n" event={"ID":"02a9089a-7dee-4192-b1c3-3ccc297c8d4f","Type":"ContainerDied","Data":"b7dff37044ce76116760ee47170d9a6dff6d02291fe8cd690a5129aa36f40a3a"}
Nov 03 00:32:32.661677 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:32.661598    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:32:32.762142 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:32.762102    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-model-cache\") pod \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\" (UID: \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\") "
Nov 03 00:32:32.762355 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:32.762156    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-home\" (UniqueName: \"kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-home\") pod \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\" (UID: \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\") "
Nov 03 00:32:32.762355 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:32.762234    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-kube-api-access-9r26w\" (UniqueName: \"kubernetes.io/projected/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-kube-api-access-9r26w\") pod \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\" (UID: \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\") "
Nov 03 00:32:32.762503 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:32.762380    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-tls-certs\" (UniqueName: \"kubernetes.io/secret/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-tls-certs\") pod \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\" (UID: \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\") "
Nov 03 00:32:32.762503 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:32.762454    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-model-cache" pod "02a9089a-7dee-4192-b1c3-3ccc297c8d4f" (UID: "02a9089a-7dee-4192-b1c3-3ccc297c8d4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:32.762594 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:32.762518    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-home" pod "02a9089a-7dee-4192-b1c3-3ccc297c8d4f" (UID: "02a9089a-7dee-4192-b1c3-3ccc297c8d4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:32.762594 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:32.762556    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-tmp-dir\") pod \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\" (UID: \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\") "
Nov 03 00:32:32.762594 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:32.762590    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-dshm\") pod \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\" (UID: \"02a9089a-7dee-4192-b1c3-3ccc297c8d4f\") "
Nov 03 00:32:32.762798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:32.762771    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-tmp-dir" pod "02a9089a-7dee-4192-b1c3-3ccc297c8d4f" (UID: "02a9089a-7dee-4192-b1c3-3ccc297c8d4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:32.763150 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:32.763125    5012 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-tmp-dir\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:32.763150 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:32.763149    5012 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-model-cache\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:32.763364 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:32.763161    5012 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-home\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:32.764971 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:32.764940    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-dshm" pod "02a9089a-7dee-4192-b1c3-3ccc297c8d4f" (UID: "02a9089a-7dee-4192-b1c3-3ccc297c8d4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:32.765207 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:32.765179    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-tls-certs" pod "02a9089a-7dee-4192-b1c3-3ccc297c8d4f" (UID: "02a9089a-7dee-4192-b1c3-3ccc297c8d4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:32:32.765303 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:32.765222    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-kube-api-access-9r26w" pod "02a9089a-7dee-4192-b1c3-3ccc297c8d4f" (UID: "02a9089a-7dee-4192-b1c3-3ccc297c8d4f"). InnerVolumeSpecName "kube-api-access-9r26w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:32.864829 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:32.864787    5012 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-tls-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:32.864829 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:32.864825    5012 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-dshm\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:32.864829 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:32.864834    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r26w\" (UniqueName: \"kubernetes.io/projected/02a9089a-7dee-4192-b1c3-3ccc297c8d4f-kube-api-access-9r26w\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:32:33.246565 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:33.246526    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ac19255-340c-4891-85d4-1b895e2d3641" path="/var/lib/kubelet/pods/9ac19255-340c-4891-85d4-1b895e2d3641/volumes"
Nov 03 00:32:33.247090 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:33.247074    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2570d35-c9ee-4069-85c2-361cba7c27fb" path="/var/lib/kubelet/pods/b2570d35-c9ee-4069-85c2-361cba7c27fb/volumes"
Nov 03 00:32:33.599607 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:33.599574    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"
Nov 03 00:32:33.600112 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:33.599608    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n" event={"ID":"02a9089a-7dee-4192-b1c3-3ccc297c8d4f","Type":"ContainerDied","Data":"42a7c45ace38f841dde842f67cfac952354fbae7cacae928531e9078b8caff1a"}
Nov 03 00:32:33.600112 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:33.599664    5012 scope.go:122] "RemoveContainer" containerID="b7dff37044ce76116760ee47170d9a6dff6d02291fe8cd690a5129aa36f40a3a"
Nov 03 00:32:33.652242 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:33.652205    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"]
Nov 03 00:32:33.662201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:33.662170    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-4ae722/praxis-route-4ae722-kserve-84d945f5f-clg9n"]
Nov 03 00:32:35.245247 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:35.245206    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02a9089a-7dee-4192-b1c3-3ccc297c8d4f" path="/var/lib/kubelet/pods/02a9089a-7dee-4192-b1c3-3ccc297c8d4f/volumes"
Nov 03 00:32:36.367473 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.367430    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"]
Nov 03 00:32:36.368298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368271    5012 cpu_manager.go:414] "containerMap: removing container" podUID="5e8e36a8-3445-41ab-a889-c529c3e707c9" containerName="istio-proxy"
Nov 03 00:32:36.368298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368299    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e8e36a8-3445-41ab-a889-c529c3e707c9" containerName="istio-proxy"
Nov 03 00:32:36.368509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368315    5012 cpu_manager.go:414] "containerMap: removing container" podUID="1f594be4-6f2b-48f1-9fef-34a19645c6dc" containerName="payload-pre-processing"
Nov 03 00:32:36.368509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368324    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f594be4-6f2b-48f1-9fef-34a19645c6dc" containerName="payload-pre-processing"
Nov 03 00:32:36.368509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368350    5012 cpu_manager.go:414] "containerMap: removing container" podUID="0528abca-dac8-491c-b6ab-d183adf6bf8a" containerName="payload-processing"
Nov 03 00:32:36.368509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368356    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0528abca-dac8-491c-b6ab-d183adf6bf8a" containerName="payload-processing"
Nov 03 00:32:36.368509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368378    5012 cpu_manager.go:414] "containerMap: removing container" podUID="8760ca6d-230b-4d95-ae34-d9384b441607" containerName="istio-proxy"
Nov 03 00:32:36.368509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368395    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8760ca6d-230b-4d95-ae34-d9384b441607" containerName="istio-proxy"
Nov 03 00:32:36.368509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368403    5012 cpu_manager.go:414] "containerMap: removing container" podUID="02a9089a-7dee-4192-b1c3-3ccc297c8d4f" containerName="main"
Nov 03 00:32:36.368509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368407    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02a9089a-7dee-4192-b1c3-3ccc297c8d4f" containerName="main"
Nov 03 00:32:36.368509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368417    5012 cpu_manager.go:414] "containerMap: removing container" podUID="72825023-1f03-47a3-a213-4cf1656e108a" containerName="payload-pre-processing"
Nov 03 00:32:36.368509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368422    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72825023-1f03-47a3-a213-4cf1656e108a" containerName="payload-pre-processing"
Nov 03 00:32:36.368509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368435    5012 cpu_manager.go:414] "containerMap: removing container" podUID="a1d8aac2-8d32-4e6f-a342-d2e2c7425f18" containerName="maas-api"
Nov 03 00:32:36.368509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368441    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1d8aac2-8d32-4e6f-a342-d2e2c7425f18" containerName="maas-api"
Nov 03 00:32:36.368509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368458    5012 cpu_manager.go:414] "containerMap: removing container" podUID="90b16caf-d83f-4c7b-a1fc-bdcb7d061653" containerName="istio-proxy"
Nov 03 00:32:36.368509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368463    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90b16caf-d83f-4c7b-a1fc-bdcb7d061653" containerName="istio-proxy"
Nov 03 00:32:36.368509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368469    5012 cpu_manager.go:414] "containerMap: removing container" podUID="b2570d35-c9ee-4069-85c2-361cba7c27fb" containerName="payload-processing"
Nov 03 00:32:36.368509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368473    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2570d35-c9ee-4069-85c2-361cba7c27fb" containerName="payload-processing"
Nov 03 00:32:36.368509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368487    5012 cpu_manager.go:414] "containerMap: removing container" podUID="7404d00f-242b-4f97-a9c6-bb04c79d7348" containerName="payload-pre-processing"
Nov 03 00:32:36.368509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368493    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7404d00f-242b-4f97-a9c6-bb04c79d7348" containerName="payload-pre-processing"
Nov 03 00:32:36.368509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368506    5012 cpu_manager.go:414] "containerMap: removing container" podUID="83f374c7-f0aa-46c2-beca-544aaf65a7b7" containerName="payload-processing"
Nov 03 00:32:36.368509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368513    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f374c7-f0aa-46c2-beca-544aaf65a7b7" containerName="payload-processing"
Nov 03 00:32:36.368509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368524    5012 cpu_manager.go:414] "containerMap: removing container" podUID="9ac19255-340c-4891-85d4-1b895e2d3641" containerName="payload-pre-processing"
Nov 03 00:32:36.370456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368532    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac19255-340c-4891-85d4-1b895e2d3641" containerName="payload-pre-processing"
Nov 03 00:32:36.370456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368546    5012 cpu_manager.go:414] "containerMap: removing container" podUID="2f8b0c71-4f60-47e3-9a22-7f25e85b690c" containerName="istio-proxy"
Nov 03 00:32:36.370456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368553    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f8b0c71-4f60-47e3-9a22-7f25e85b690c" containerName="istio-proxy"
Nov 03 00:32:36.370456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368682    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="8760ca6d-230b-4d95-ae34-d9384b441607" containerName="istio-proxy"
Nov 03 00:32:36.370456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368693    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="02a9089a-7dee-4192-b1c3-3ccc297c8d4f" containerName="main"
Nov 03 00:32:36.370456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368700    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="72825023-1f03-47a3-a213-4cf1656e108a" containerName="payload-pre-processing"
Nov 03 00:32:36.370456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368710    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e8e36a8-3445-41ab-a889-c529c3e707c9" containerName="istio-proxy"
Nov 03 00:32:36.370456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368716    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1d8aac2-8d32-4e6f-a342-d2e2c7425f18" containerName="maas-api"
Nov 03 00:32:36.370456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368724    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="90b16caf-d83f-4c7b-a1fc-bdcb7d061653" containerName="istio-proxy"
Nov 03 00:32:36.370456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368730    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2570d35-c9ee-4069-85c2-361cba7c27fb" containerName="payload-processing"
Nov 03 00:32:36.370456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368735    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="7404d00f-242b-4f97-a9c6-bb04c79d7348" containerName="payload-pre-processing"
Nov 03 00:32:36.370456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368739    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="83f374c7-f0aa-46c2-beca-544aaf65a7b7" containerName="payload-processing"
Nov 03 00:32:36.370456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368745    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ac19255-340c-4891-85d4-1b895e2d3641" containerName="payload-pre-processing"
Nov 03 00:32:36.370456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368749    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f8b0c71-4f60-47e3-9a22-7f25e85b690c" containerName="istio-proxy"
Nov 03 00:32:36.370456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368758    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="90b16caf-d83f-4c7b-a1fc-bdcb7d061653" containerName="istio-proxy"
Nov 03 00:32:36.370456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368768    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f594be4-6f2b-48f1-9fef-34a19645c6dc" containerName="payload-pre-processing"
Nov 03 00:32:36.370456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.368776    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="0528abca-dac8-491c-b6ab-d183adf6bf8a" containerName="payload-processing"
Nov 03 00:32:36.374258 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.374233    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.390321 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.387404    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-5d51a5-openshift-default-dockercfg-wxxrd\""
Nov 03 00:32:36.403247 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.403215    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"]
Nov 03 00:32:36.464601 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.464567    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-workload-socket\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.464772 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.464606    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-token\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.464772 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.464750    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-credential-socket\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.464846 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.464785    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-podinfo\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.464935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.464891    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-envoy\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.464989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.464955    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-workload-certs\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.465053 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.465040    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istiod-ca-cert\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.465089 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.465063    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69stt\" (UniqueName: \"kubernetes.io/projected/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-kube-api-access-69stt\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.465127 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.465098    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-data\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.566868 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.566827    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-workload-socket\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.567150 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.567130    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-token\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.567390 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.567372    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-credential-socket\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.567545 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.567526    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-podinfo\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.567749 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.567731    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-envoy\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.567956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.567939    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-workload-certs\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.568145 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.568128    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istiod-ca-cert\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.568263 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.568248    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69stt\" (UniqueName: \"kubernetes.io/projected/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-kube-api-access-69stt\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.568412 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.568398    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-data\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.569058 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.569036    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-data\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.569489 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.569470    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-workload-socket\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.570211 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.570192    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-credential-socket\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.576518 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.576229    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-workload-certs\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.581211 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.579038    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-envoy\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.581211 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.579689    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istiod-ca-cert\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.593513 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.593304    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-podinfo\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.594957 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.594221    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-token\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.600805 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.600767    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69stt\" (UniqueName: \"kubernetes.io/projected/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-kube-api-access-69stt\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:36.691424 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:36.691278    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:37.011380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:37.011340    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"]
Nov 03 00:32:37.012524 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:32:37.012495    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f45c1d_a84e_4e9b_bec1_6df5d0d1eab1.slice/crio-03bd0adf1baf4dcd9d784a44c648a3ae94d99367ccc1ef768b6bc5d153df4962 WatchSource:0}: Error finding container 03bd0adf1baf4dcd9d784a44c648a3ae94d99367ccc1ef768b6bc5d153df4962: Status 404 returned error can't find the container with id 03bd0adf1baf4dcd9d784a44c648a3ae94d99367ccc1ef768b6bc5d153df4962
Nov 03 00:32:37.015263 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:37.015227    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:32:37.015357 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:37.015300    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:32:37.015357 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:37.015333    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:32:37.367233 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:32:37.367198    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:32:37.659890 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:37.659687    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5" event={"ID":"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1","Type":"ContainerStarted","Data":"0ddf9e2e137defc5edcc56e8d54690b8a3da2d0491340c0c8d72a41fba6c4332"}
Nov 03 00:32:37.659890 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:37.659818    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5" event={"ID":"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1","Type":"ContainerStarted","Data":"03bd0adf1baf4dcd9d784a44c648a3ae94d99367ccc1ef768b6bc5d153df4962"}
Nov 03 00:32:37.700593 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:37.700509    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:38.698593 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:38.698558    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:38.757550 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:38.757486    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5" podStartSLOduration=2.757470267 podStartE2EDuration="2.757470267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:32:37.723176311 +0000 UTC m=+1640.602760356" watchObservedRunningTime="2026-11-03 00:32:38.757470267 +0000 UTC m=+1641.637054311"
Nov 03 00:32:39.679095 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:39.679059    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:39.680160 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:39.680140    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:32:43.028919 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:43.028887    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c8e7371-7s2q4"]
Nov 03 00:32:43.042989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:43.042196    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c8e7371-7s2q4"]
Nov 03 00:32:43.244793 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:43.244757    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4696ff14-ca42-478f-b026-6976d577cd88" path="/var/lib/kubelet/pods/4696ff14-ca42-478f-b026-6976d577cd88/volumes"
Nov 03 00:32:54.050791 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:54.050749    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp"]
Nov 03 00:32:54.341373 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:54.341288    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk_6029bbe2-1d00-4960-8555-f5e6e1588bfa/payload-processing/0.log"
Nov 03 00:32:54.552649 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:54.552612    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-885tl_df410080-3e08-4f8d-b1d3-84db851d8782/payload-processing/0.log"
Nov 03 00:32:54.763491 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:54.763461    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-885tl_df410080-3e08-4f8d-b1d3-84db851d8782/payload-processing/0.log"
Nov 03 00:32:54.967329 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:54.967302    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr_fb399f46-4a86-4660-bfef-c6267eaff189/payload-processing/0.log"
Nov 03 00:32:55.155337 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:55.155309    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr_fb399f46-4a86-4660-bfef-c6267eaff189/payload-processing/0.log"
Nov 03 00:32:59.051416 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:32:59.051331    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp" podUID="012c331f-034b-46f2-a430-ca3d7cdc9e4b" containerName="payload-processing" containerID="cri-o://606390c165991891bdc50130d6f6ec8d3b9d84299755637ba17d2622101a38af" gracePeriod=25
Nov 03 00:33:01.111182 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:01.111136    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"]
Nov 03 00:33:04.768439 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:04.768394    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp" podUID="012c331f-034b-46f2-a430-ca3d7cdc9e4b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:33:05.602422 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.602361    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"]
Nov 03 00:33:05.604371 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.604343    5012 cpu_manager.go:414] "containerMap: removing container" podUID="5e8e36a8-3445-41ab-a889-c529c3e707c9" containerName="istio-proxy"
Nov 03 00:33:05.604371 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.604369    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e8e36a8-3445-41ab-a889-c529c3e707c9" containerName="istio-proxy"
Nov 03 00:33:05.604591 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.604439    5012 cpu_manager.go:414] "containerMap: removing container" podUID="90b16caf-d83f-4c7b-a1fc-bdcb7d061653" containerName="istio-proxy"
Nov 03 00:33:05.604591 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.604448    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90b16caf-d83f-4c7b-a1fc-bdcb7d061653" containerName="istio-proxy"
Nov 03 00:33:05.604697 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.604596    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e8e36a8-3445-41ab-a889-c529c3e707c9" containerName="istio-proxy"
Nov 03 00:33:05.608535 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.608513    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.657664 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.657624    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8797w\" (UniqueName: \"kubernetes.io/projected/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-kube-api-access-8797w\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.657969 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.657942    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.658199 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.658176    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.658354 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.658337    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.658498 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.658480    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.658612 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.658596    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.658728 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.658711    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.658924 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.658890    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.659141 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.659104    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.663174 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.663137    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"]
Nov 03 00:33:05.762934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.762776    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.762934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.762832    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.762934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.762863    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.762934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.762894    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.763445 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.763414    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.763727 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.763705    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.763774 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.763759    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.763891 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.763874    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.763986 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.763948    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8797w\" (UniqueName: \"kubernetes.io/projected/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-kube-api-access-8797w\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.764024 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.764010    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.764063 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.764016    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.764140 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.764121    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.764443 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.764419    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.765164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.765140    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.766521 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.766495    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.767352 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.767335    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.809933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.808132    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.811966 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.811866    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8797w\" (UniqueName: \"kubernetes.io/projected/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-kube-api-access-8797w\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:05.927825 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:05.927726    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:06.268828 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:33:06.268795    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c86d7ef_71cd_4712_b0ed_5b1e4c50d3b9.slice/crio-2e05a4e17b07e5a6c3577643144970356b47b4df4c075bcb7ba726c7a53f9738 WatchSource:0}: Error finding container 2e05a4e17b07e5a6c3577643144970356b47b4df4c075bcb7ba726c7a53f9738: Status 404 returned error can't find the container with id 2e05a4e17b07e5a6c3577643144970356b47b4df4c075bcb7ba726c7a53f9738
Nov 03 00:33:06.269006 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:06.268981    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"]
Nov 03 00:33:06.271274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:06.271236    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:06.271373 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:06.271307    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:06.271373 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:06.271338    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:06.985926 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:06.985876    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6" event={"ID":"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9","Type":"ContainerStarted","Data":"bebd93a441dd346b3ad67f75f901e29628a42001f82a11f6f733e61424ea621f"}
Nov 03 00:33:06.986332 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:06.985982    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6" event={"ID":"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9","Type":"ContainerStarted","Data":"2e05a4e17b07e5a6c3577643144970356b47b4df4c075bcb7ba726c7a53f9738"}
Nov 03 00:33:07.046637 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:07.046583    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6" podStartSLOduration=2.046567716 podStartE2EDuration="2.046567716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:07.034283389 +0000 UTC m=+1669.913867434" watchObservedRunningTime="2026-11-03 00:33:07.046567716 +0000 UTC m=+1669.926151761"
Nov 03 00:33:07.364897 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:33:07.364862    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:33:07.928605 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:07.928571    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:07.930611 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:07.930579    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 03 00:33:07.930759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:07.930650    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6" podUID="8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 03 00:33:08.929022 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:08.928903    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 03 00:33:08.929022 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:08.928992    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6" podUID="8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 03 00:33:09.928645 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:09.928609    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 03 00:33:09.928852 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:09.928661    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6" podUID="8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 03 00:33:10.928473 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:10.928438    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 03 00:33:10.928918 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:10.928494    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6" podUID="8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 03 00:33:11.929130 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:11.929093    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 03 00:33:11.929536 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:11.929147    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6" podUID="8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 03 00:33:12.929132 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:12.929093    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 03 00:33:12.929560 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:12.929152    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6" podUID="8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 03 00:33:14.308459 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:14.308422    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:14.309000 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:14.308903    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:14.309862 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:14.309836    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:33:14.768406 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:14.768367    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp" podUID="012c331f-034b-46f2-a430-ca3d7cdc9e4b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:33:15.254928 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:15.254884    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"]
Nov 03 00:33:15.255196 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:15.255138    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8" podUID="59529c4e-6121-4fcf-94c2-4671ab16d5e2" containerName="istio-proxy" containerID="cri-o://ca5effc975fb64c83d56439a1d41011c0a670f442f520c66ba00967249c5bbe4" gracePeriod=30
Nov 03 00:33:16.137175 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:16.137137    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx" podUID="10bb726f-ca52-4d99-994d-a82e852764df" containerName="main" containerID="cri-o://745f93362bb36b735a58cecf8709c75e63c1e442c33453013c35a982cf30c457" gracePeriod=45
Nov 03 00:33:16.410199 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:16.410174    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:33:16.426973 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:16.426950    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-model-cache\") pod \"10bb726f-ca52-4d99-994d-a82e852764df\" (UID: \"10bb726f-ca52-4d99-994d-a82e852764df\") "
Nov 03 00:33:16.427082 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:16.427015    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-home\" (UniqueName: \"kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-home\") pod \"10bb726f-ca52-4d99-994d-a82e852764df\" (UID: \"10bb726f-ca52-4d99-994d-a82e852764df\") "
Nov 03 00:33:16.427131 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:16.427101    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-tmp-dir\") pod \"10bb726f-ca52-4d99-994d-a82e852764df\" (UID: \"10bb726f-ca52-4d99-994d-a82e852764df\") "
Nov 03 00:33:16.427171 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:16.427158    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10bb726f-ca52-4d99-994d-a82e852764df-tls-certs\" (UniqueName: \"kubernetes.io/secret/10bb726f-ca52-4d99-994d-a82e852764df-tls-certs\") pod \"10bb726f-ca52-4d99-994d-a82e852764df\" (UID: \"10bb726f-ca52-4d99-994d-a82e852764df\") "
Nov 03 00:33:16.427223 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:16.427207    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10bb726f-ca52-4d99-994d-a82e852764df-kube-api-access-kswtg\" (UniqueName: \"kubernetes.io/projected/10bb726f-ca52-4d99-994d-a82e852764df-kube-api-access-kswtg\") pod \"10bb726f-ca52-4d99-994d-a82e852764df\" (UID: \"10bb726f-ca52-4d99-994d-a82e852764df\") "
Nov 03 00:33:16.427261 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:16.427204    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-model-cache" pod "10bb726f-ca52-4d99-994d-a82e852764df" (UID: "10bb726f-ca52-4d99-994d-a82e852764df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:33:16.427261 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:16.427240    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-dshm\" (UniqueName: \"kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-dshm\") pod \"10bb726f-ca52-4d99-994d-a82e852764df\" (UID: \"10bb726f-ca52-4d99-994d-a82e852764df\") "
Nov 03 00:33:16.427340 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:16.427267    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-home" pod "10bb726f-ca52-4d99-994d-a82e852764df" (UID: "10bb726f-ca52-4d99-994d-a82e852764df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:33:16.427404 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:16.427364    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-tmp-dir" pod "10bb726f-ca52-4d99-994d-a82e852764df" (UID: "10bb726f-ca52-4d99-994d-a82e852764df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:33:16.427748 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:16.427726    5012 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-model-cache\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:16.427748 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:16.427748    5012 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-home\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:16.427957 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:16.427762    5012 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-tmp-dir\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:16.429389 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:16.429365    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-dshm" pod "10bb726f-ca52-4d99-994d-a82e852764df" (UID: "10bb726f-ca52-4d99-994d-a82e852764df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:33:16.429516 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:16.429459    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10bb726f-ca52-4d99-994d-a82e852764df-kube-api-access-kswtg" pod "10bb726f-ca52-4d99-994d-a82e852764df" (UID: "10bb726f-ca52-4d99-994d-a82e852764df"). InnerVolumeSpecName "kube-api-access-kswtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:33:16.429516 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:16.429468    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10bb726f-ca52-4d99-994d-a82e852764df-tls-certs" pod "10bb726f-ca52-4d99-994d-a82e852764df" (UID: "10bb726f-ca52-4d99-994d-a82e852764df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:33:16.529511 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:16.529464    5012 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10bb726f-ca52-4d99-994d-a82e852764df-tls-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:16.529511 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:16.529507    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kswtg\" (UniqueName: \"kubernetes.io/projected/10bb726f-ca52-4d99-994d-a82e852764df-kube-api-access-kswtg\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:16.529511 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:16.529519    5012 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10bb726f-ca52-4d99-994d-a82e852764df-dshm\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:17.105159 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:17.105121    5012 generic.go:356] "Generic (PLEG): container finished" podID="10bb726f-ca52-4d99-994d-a82e852764df" containerID="745f93362bb36b735a58cecf8709c75e63c1e442c33453013c35a982cf30c457" exitCode=0
Nov 03 00:33:17.105377 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:17.105177    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"
Nov 03 00:33:17.105377 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:17.105209    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx" event={"ID":"10bb726f-ca52-4d99-994d-a82e852764df","Type":"ContainerDied","Data":"745f93362bb36b735a58cecf8709c75e63c1e442c33453013c35a982cf30c457"}
Nov 03 00:33:17.105377 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:17.105255    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx" event={"ID":"10bb726f-ca52-4d99-994d-a82e852764df","Type":"ContainerDied","Data":"2e935726580fc13b6c8e2682c98266eb7287531475ed75cea696e69332cad45f"}
Nov 03 00:33:17.105377 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:17.105277    5012 scope.go:122] "RemoveContainer" containerID="745f93362bb36b735a58cecf8709c75e63c1e442c33453013c35a982cf30c457"
Nov 03 00:33:17.117131 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:17.117111    5012 scope.go:122] "RemoveContainer" containerID="745f93362bb36b735a58cecf8709c75e63c1e442c33453013c35a982cf30c457"
Nov 03 00:33:17.117395 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:33:17.117375    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"745f93362bb36b735a58cecf8709c75e63c1e442c33453013c35a982cf30c457\": container with ID starting with 745f93362bb36b735a58cecf8709c75e63c1e442c33453013c35a982cf30c457 not found: ID does not exist" containerID="745f93362bb36b735a58cecf8709c75e63c1e442c33453013c35a982cf30c457"
Nov 03 00:33:17.117465 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:17.117408    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"745f93362bb36b735a58cecf8709c75e63c1e442c33453013c35a982cf30c457"} err="failed to get container status \"745f93362bb36b735a58cecf8709c75e63c1e442c33453013c35a982cf30c457\": rpc error: code = NotFound desc = could not find container \"745f93362bb36b735a58cecf8709c75e63c1e442c33453013c35a982cf30c457\": container with ID starting with 745f93362bb36b735a58cecf8709c75e63c1e442c33453013c35a982cf30c457 not found: ID does not exist"
Nov 03 00:33:17.151511 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:17.151486    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"]
Nov 03 00:33:17.162080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:17.162055    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-a43474/ipp-route-a43474-kserve-768f6487c-xtbkx"]
Nov 03 00:33:17.255432 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:17.255396    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10bb726f-ca52-4d99-994d-a82e852764df" path="/var/lib/kubelet/pods/10bb726f-ca52-4d99-994d-a82e852764df/volumes"
Nov 03 00:33:19.339838 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:19.339815    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp"
Nov 03 00:33:19.362600 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:19.362567    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012c331f-034b-46f2-a430-ca3d7cdc9e4b-kube-api-access-zrpwh\" (UniqueName: \"kubernetes.io/projected/012c331f-034b-46f2-a430-ca3d7cdc9e4b-kube-api-access-zrpwh\") pod \"012c331f-034b-46f2-a430-ca3d7cdc9e4b\" (UID: \"012c331f-034b-46f2-a430-ca3d7cdc9e4b\") "
Nov 03 00:33:19.362771 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:19.362757    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/012c331f-034b-46f2-a430-ca3d7cdc9e4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/012c331f-034b-46f2-a430-ca3d7cdc9e4b-plugins-config-volume\") pod \"012c331f-034b-46f2-a430-ca3d7cdc9e4b\" (UID: \"012c331f-034b-46f2-a430-ca3d7cdc9e4b\") "
Nov 03 00:33:19.363180 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:19.363154    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012c331f-034b-46f2-a430-ca3d7cdc9e4b-plugins-config-volume" pod "012c331f-034b-46f2-a430-ca3d7cdc9e4b" (UID: "012c331f-034b-46f2-a430-ca3d7cdc9e4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:33:19.363292 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:19.363252    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/012c331f-034b-46f2-a430-ca3d7cdc9e4b-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:19.364667 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:19.364644    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012c331f-034b-46f2-a430-ca3d7cdc9e4b-kube-api-access-zrpwh" pod "012c331f-034b-46f2-a430-ca3d7cdc9e4b" (UID: "012c331f-034b-46f2-a430-ca3d7cdc9e4b"). InnerVolumeSpecName "kube-api-access-zrpwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:33:19.465047 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:19.465012    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrpwh\" (UniqueName: \"kubernetes.io/projected/012c331f-034b-46f2-a430-ca3d7cdc9e4b-kube-api-access-zrpwh\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:20.134586 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.134549    5012 generic.go:356] "Generic (PLEG): container finished" podID="012c331f-034b-46f2-a430-ca3d7cdc9e4b" containerID="606390c165991891bdc50130d6f6ec8d3b9d84299755637ba17d2622101a38af" exitCode=0
Nov 03 00:33:20.134762 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.134591    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp"
Nov 03 00:33:20.134762 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.134634    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp" event={"ID":"012c331f-034b-46f2-a430-ca3d7cdc9e4b","Type":"ContainerDied","Data":"606390c165991891bdc50130d6f6ec8d3b9d84299755637ba17d2622101a38af"}
Nov 03 00:33:20.134762 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.134684    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp" event={"ID":"012c331f-034b-46f2-a430-ca3d7cdc9e4b","Type":"ContainerDied","Data":"2c26ffa62d03044f8f3c4663108a32cf065fd8ac506f1087cc1c4edf0267b0f2"}
Nov 03 00:33:20.134762 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.134716    5012 scope.go:122] "RemoveContainer" containerID="606390c165991891bdc50130d6f6ec8d3b9d84299755637ba17d2622101a38af"
Nov 03 00:33:20.145471 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.145444    5012 scope.go:122] "RemoveContainer" containerID="606390c165991891bdc50130d6f6ec8d3b9d84299755637ba17d2622101a38af"
Nov 03 00:33:20.145790 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:33:20.145762    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"606390c165991891bdc50130d6f6ec8d3b9d84299755637ba17d2622101a38af\": container with ID starting with 606390c165991891bdc50130d6f6ec8d3b9d84299755637ba17d2622101a38af not found: ID does not exist" containerID="606390c165991891bdc50130d6f6ec8d3b9d84299755637ba17d2622101a38af"
Nov 03 00:33:20.145891 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.145795    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"606390c165991891bdc50130d6f6ec8d3b9d84299755637ba17d2622101a38af"} err="failed to get container status \"606390c165991891bdc50130d6f6ec8d3b9d84299755637ba17d2622101a38af\": rpc error: code = NotFound desc = could not find container \"606390c165991891bdc50130d6f6ec8d3b9d84299755637ba17d2622101a38af\": container with ID starting with 606390c165991891bdc50130d6f6ec8d3b9d84299755637ba17d2622101a38af not found: ID does not exist"
Nov 03 00:33:20.183471 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.183442    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp"]
Nov 03 00:33:20.196524 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.196501    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4ae722-68bf5d9f85-ljbnp"]
Nov 03 00:33:20.563539 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.563513    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:33:20.683726 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.683692    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-credential-socket\") pod \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") "
Nov 03 00:33:20.683888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.683768    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-token\" (UniqueName: \"kubernetes.io/projected/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-token\") pod \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") "
Nov 03 00:33:20.683888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.683799    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-podinfo\") pod \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") "
Nov 03 00:33:20.683888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.683842    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-data\") pod \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") "
Nov 03 00:33:20.683888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.683881    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-workload-certs\") pod \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") "
Nov 03 00:33:20.684073 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.683972    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istiod-ca-cert\") pod \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") "
Nov 03 00:33:20.684073 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.684015    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-envoy\") pod \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") "
Nov 03 00:33:20.684073 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.684067    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59529c4e-6121-4fcf-94c2-4671ab16d5e2-kube-api-access-ndstc\" (UniqueName: \"kubernetes.io/projected/59529c4e-6121-4fcf-94c2-4671ab16d5e2-kube-api-access-ndstc\") pod \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") "
Nov 03 00:33:20.684243 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.684148    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-workload-socket\") pod \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\" (UID: \"59529c4e-6121-4fcf-94c2-4671ab16d5e2\") "
Nov 03 00:33:20.684314 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.684206    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-workload-certs" pod "59529c4e-6121-4fcf-94c2-4671ab16d5e2" (UID: "59529c4e-6121-4fcf-94c2-4671ab16d5e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:33:20.684314 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.684242    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-data" pod "59529c4e-6121-4fcf-94c2-4671ab16d5e2" (UID: "59529c4e-6121-4fcf-94c2-4671ab16d5e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:33:20.684435 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.684320    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-credential-socket" pod "59529c4e-6121-4fcf-94c2-4671ab16d5e2" (UID: "59529c4e-6121-4fcf-94c2-4671ab16d5e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:33:20.684435 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.684387    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istiod-ca-cert" pod "59529c4e-6121-4fcf-94c2-4671ab16d5e2" (UID: "59529c4e-6121-4fcf-94c2-4671ab16d5e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:33:20.684591 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.684570    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-workload-socket" pod "59529c4e-6121-4fcf-94c2-4671ab16d5e2" (UID: "59529c4e-6121-4fcf-94c2-4671ab16d5e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:33:20.684861 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.684840    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:20.684983 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.684863    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:20.684983 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.684877    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:20.684983 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.684890    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:20.684983 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.684903    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:20.686346 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.686323    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-podinfo" pod "59529c4e-6121-4fcf-94c2-4671ab16d5e2" (UID: "59529c4e-6121-4fcf-94c2-4671ab16d5e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:33:20.686575 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.686536    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-envoy" pod "59529c4e-6121-4fcf-94c2-4671ab16d5e2" (UID: "59529c4e-6121-4fcf-94c2-4671ab16d5e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:33:20.686693 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.686660    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-token" pod "59529c4e-6121-4fcf-94c2-4671ab16d5e2" (UID: "59529c4e-6121-4fcf-94c2-4671ab16d5e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:33:20.687243 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.687225    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59529c4e-6121-4fcf-94c2-4671ab16d5e2-kube-api-access-ndstc" pod "59529c4e-6121-4fcf-94c2-4671ab16d5e2" (UID: "59529c4e-6121-4fcf-94c2-4671ab16d5e2"). InnerVolumeSpecName "kube-api-access-ndstc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:33:20.786128 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.786097    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:20.786128 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.786127    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndstc\" (UniqueName: \"kubernetes.io/projected/59529c4e-6121-4fcf-94c2-4671ab16d5e2-kube-api-access-ndstc\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:20.786128 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.786137    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:20.786369 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:20.786146    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59529c4e-6121-4fcf-94c2-4671ab16d5e2-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:21.143611 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:21.143575    5012 generic.go:356] "Generic (PLEG): container finished" podID="59529c4e-6121-4fcf-94c2-4671ab16d5e2" containerID="ca5effc975fb64c83d56439a1d41011c0a670f442f520c66ba00967249c5bbe4" exitCode=0
Nov 03 00:33:21.143802 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:21.143610    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8" event={"ID":"59529c4e-6121-4fcf-94c2-4671ab16d5e2","Type":"ContainerDied","Data":"ca5effc975fb64c83d56439a1d41011c0a670f442f520c66ba00967249c5bbe4"}
Nov 03 00:33:21.143802 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:21.143635    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"
Nov 03 00:33:21.143802 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:21.143668    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8" event={"ID":"59529c4e-6121-4fcf-94c2-4671ab16d5e2","Type":"ContainerDied","Data":"7b833565be823a8dd8973cefd8c9c5e700b06714147b1cc7dd4cecd913fdd117"}
Nov 03 00:33:21.143802 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:21.143694    5012 scope.go:122] "RemoveContainer" containerID="ca5effc975fb64c83d56439a1d41011c0a670f442f520c66ba00967249c5bbe4"
Nov 03 00:33:21.155039 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:21.155021    5012 scope.go:122] "RemoveContainer" containerID="1571366599c84892f1f34fef6db8205a271755c5d8e094464cf57108351e22d2"
Nov 03 00:33:21.164592 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:21.164570    5012 scope.go:122] "RemoveContainer" containerID="ca5effc975fb64c83d56439a1d41011c0a670f442f520c66ba00967249c5bbe4"
Nov 03 00:33:21.164850 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:33:21.164829    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca5effc975fb64c83d56439a1d41011c0a670f442f520c66ba00967249c5bbe4\": container with ID starting with ca5effc975fb64c83d56439a1d41011c0a670f442f520c66ba00967249c5bbe4 not found: ID does not exist" containerID="ca5effc975fb64c83d56439a1d41011c0a670f442f520c66ba00967249c5bbe4"
Nov 03 00:33:21.164923 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:21.164858    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca5effc975fb64c83d56439a1d41011c0a670f442f520c66ba00967249c5bbe4"} err="failed to get container status \"ca5effc975fb64c83d56439a1d41011c0a670f442f520c66ba00967249c5bbe4\": rpc error: code = NotFound desc = could not find container \"ca5effc975fb64c83d56439a1d41011c0a670f442f520c66ba00967249c5bbe4\": container with ID starting with ca5effc975fb64c83d56439a1d41011c0a670f442f520c66ba00967249c5bbe4 not found: ID does not exist"
Nov 03 00:33:21.164923 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:21.164875    5012 scope.go:122] "RemoveContainer" containerID="1571366599c84892f1f34fef6db8205a271755c5d8e094464cf57108351e22d2"
Nov 03 00:33:21.165120 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:33:21.165097    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1571366599c84892f1f34fef6db8205a271755c5d8e094464cf57108351e22d2\": container with ID starting with 1571366599c84892f1f34fef6db8205a271755c5d8e094464cf57108351e22d2 not found: ID does not exist" containerID="1571366599c84892f1f34fef6db8205a271755c5d8e094464cf57108351e22d2"
Nov 03 00:33:21.165176 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:21.165123    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1571366599c84892f1f34fef6db8205a271755c5d8e094464cf57108351e22d2"} err="failed to get container status \"1571366599c84892f1f34fef6db8205a271755c5d8e094464cf57108351e22d2\": rpc error: code = NotFound desc = could not find container \"1571366599c84892f1f34fef6db8205a271755c5d8e094464cf57108351e22d2\": container with ID starting with 1571366599c84892f1f34fef6db8205a271755c5d8e094464cf57108351e22d2 not found: ID does not exist"
Nov 03 00:33:21.202507 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:21.202483    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"]
Nov 03 00:33:21.212725 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:21.212703    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-4ae722-openshift-default-9f7cdddd9-mv6g8"]
Nov 03 00:33:21.244585 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:21.244547    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="012c331f-034b-46f2-a430-ca3d7cdc9e4b" path="/var/lib/kubelet/pods/012c331f-034b-46f2-a430-ca3d7cdc9e4b/volumes"
Nov 03 00:33:21.245169 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:21.245147    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59529c4e-6121-4fcf-94c2-4671ab16d5e2" path="/var/lib/kubelet/pods/59529c4e-6121-4fcf-94c2-4671ab16d5e2/volumes"
Nov 03 00:33:27.511774 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.511736    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"]
Nov 03 00:33:27.512706 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.512682    5012 cpu_manager.go:414] "containerMap: removing container" podUID="59529c4e-6121-4fcf-94c2-4671ab16d5e2" containerName="istio-proxy"
Nov 03 00:33:27.512753 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.512711    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59529c4e-6121-4fcf-94c2-4671ab16d5e2" containerName="istio-proxy"
Nov 03 00:33:27.512790 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.512773    5012 cpu_manager.go:414] "containerMap: removing container" podUID="10bb726f-ca52-4d99-994d-a82e852764df" containerName="main"
Nov 03 00:33:27.512790 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.512781    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10bb726f-ca52-4d99-994d-a82e852764df" containerName="main"
Nov 03 00:33:27.512854 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.512817    5012 cpu_manager.go:414] "containerMap: removing container" podUID="012c331f-034b-46f2-a430-ca3d7cdc9e4b" containerName="payload-processing"
Nov 03 00:33:27.512854 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.512825    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="012c331f-034b-46f2-a430-ca3d7cdc9e4b" containerName="payload-processing"
Nov 03 00:33:27.513026 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.513010    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="10bb726f-ca52-4d99-994d-a82e852764df" containerName="main"
Nov 03 00:33:27.513071 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.513035    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="012c331f-034b-46f2-a430-ca3d7cdc9e4b" containerName="payload-processing"
Nov 03 00:33:27.513071 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.513047    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="59529c4e-6121-4fcf-94c2-4671ab16d5e2" containerName="istio-proxy"
Nov 03 00:33:27.513071 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.513058    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="59529c4e-6121-4fcf-94c2-4671ab16d5e2" containerName="istio-proxy"
Nov 03 00:33:27.518800 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.518777    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.545276 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.545247    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-a19e67-openshift-default-dockercfg-pqnsw\""
Nov 03 00:33:27.549742 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.549714    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"]
Nov 03 00:33:27.603384 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.603339    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-token\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.603555 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.603502    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-data\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.603648 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.603617    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-credential-socket\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.603738 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.603723    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istiod-ca-cert\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.603783 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.603769    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-envoy\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.603820 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.603789    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-podinfo\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.603820 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.603812    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-workload-socket\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.604020 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.603996    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfcjq\" (UniqueName: \"kubernetes.io/projected/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-kube-api-access-zfcjq\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.604101 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.604065    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-workload-certs\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.727669 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.727621    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfcjq\" (UniqueName: \"kubernetes.io/projected/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-kube-api-access-zfcjq\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.728643 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.728608    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-workload-certs\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.728777 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.728679    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-token\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.728860 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.728783    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-data\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.728964 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.728874    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-credential-socket\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.728964 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.728952    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istiod-ca-cert\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.729088 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.729003    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-envoy\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.729088 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.729031    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-podinfo\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.729088 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.729063    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-workload-socket\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.729590 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.729560    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-workload-socket\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.729855 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.729832    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-workload-certs\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.731383 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.730701    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istiod-ca-cert\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.731383 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.730970    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-data\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.731383 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.731189    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-credential-socket\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.755932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.742547    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-podinfo\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.755932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.750893    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-token\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.769988 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.762221    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-envoy\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.769988 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.763746    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfcjq\" (UniqueName: \"kubernetes.io/projected/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-kube-api-access-zfcjq\") pod \"e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") " pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:27.837242 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:27.837203    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:28.155456 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:33:28.155423    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ddb91bd_5e27_43f9_bc78_e9c27ba02ac5.slice/crio-f9de7a9a0a5e7b8946aeb67b69e1ae5a187d299de3ee9392d339aac565958673 WatchSource:0}: Error finding container f9de7a9a0a5e7b8946aeb67b69e1ae5a187d299de3ee9392d339aac565958673: Status 404 returned error can't find the container with id f9de7a9a0a5e7b8946aeb67b69e1ae5a187d299de3ee9392d339aac565958673
Nov 03 00:33:28.155777 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:28.155755    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"]
Nov 03 00:33:28.157773 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:28.157743    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:28.157866 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:28.157807    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:28.157866 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:28.157836    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:28.204064 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:28.204034    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln" event={"ID":"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5","Type":"ContainerStarted","Data":"f9de7a9a0a5e7b8946aeb67b69e1ae5a187d299de3ee9392d339aac565958673"}
Nov 03 00:33:29.214800 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:29.214766    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln" event={"ID":"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5","Type":"ContainerStarted","Data":"45a709ea2c77440cc22cf716ce9e4b2d0eeefe0c48ed89ab9faa359d04fbffd3"}
Nov 03 00:33:29.278572 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:29.278507    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln" podStartSLOduration=2.278489243 podStartE2EDuration="2.278489243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:29.261703511 +0000 UTC m=+1692.141287553" watchObservedRunningTime="2026-11-03 00:33:29.278489243 +0000 UTC m=+1692.158073287"
Nov 03 00:33:29.840253 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:29.839957    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:29.896364 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:29.896329    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:30.224457 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:30.224425    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:30.225461 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:30.225438    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:33:37.365345 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:33:37.365304    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:33:37.470309 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:37.470265    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v"]
Nov 03 00:33:37.470553 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:37.470523    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v" podUID="907fc1f1-9a24-4941-a792-5d70a3bb3566" containerName="maas-api" containerID="cri-o://109647a616b4738d2f4051e01835ea8c4a1073ccd8f5ac33d00de9404d56b195" gracePeriod=30
Nov 03 00:33:37.763055 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:37.762995    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v"
Nov 03 00:33:37.832049 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:37.830890    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/907fc1f1-9a24-4941-a792-5d70a3bb3566-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/907fc1f1-9a24-4941-a792-5d70a3bb3566-maas-api-tls\") pod \"907fc1f1-9a24-4941-a792-5d70a3bb3566\" (UID: \"907fc1f1-9a24-4941-a792-5d70a3bb3566\") "
Nov 03 00:33:37.832049 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:37.831009    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/907fc1f1-9a24-4941-a792-5d70a3bb3566-kube-api-access-qg75k\" (UniqueName: \"kubernetes.io/projected/907fc1f1-9a24-4941-a792-5d70a3bb3566-kube-api-access-qg75k\") pod \"907fc1f1-9a24-4941-a792-5d70a3bb3566\" (UID: \"907fc1f1-9a24-4941-a792-5d70a3bb3566\") "
Nov 03 00:33:37.832049 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:37.831065    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/907fc1f1-9a24-4941-a792-5d70a3bb3566-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/907fc1f1-9a24-4941-a792-5d70a3bb3566-openshift-service-ca\") pod \"907fc1f1-9a24-4941-a792-5d70a3bb3566\" (UID: \"907fc1f1-9a24-4941-a792-5d70a3bb3566\") "
Nov 03 00:33:37.832049 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:37.831110    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/907fc1f1-9a24-4941-a792-5d70a3bb3566-metrics-certs\" (UniqueName: \"kubernetes.io/secret/907fc1f1-9a24-4941-a792-5d70a3bb3566-metrics-certs\") pod \"907fc1f1-9a24-4941-a792-5d70a3bb3566\" (UID: \"907fc1f1-9a24-4941-a792-5d70a3bb3566\") "
Nov 03 00:33:37.839385 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:37.839089    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/907fc1f1-9a24-4941-a792-5d70a3bb3566-maas-api-tls" pod "907fc1f1-9a24-4941-a792-5d70a3bb3566" (UID: "907fc1f1-9a24-4941-a792-5d70a3bb3566"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:33:37.844813 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:37.843315    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/907fc1f1-9a24-4941-a792-5d70a3bb3566-openshift-service-ca" pod "907fc1f1-9a24-4941-a792-5d70a3bb3566" (UID: "907fc1f1-9a24-4941-a792-5d70a3bb3566"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:33:37.844813 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:37.843391    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/907fc1f1-9a24-4941-a792-5d70a3bb3566-metrics-certs" pod "907fc1f1-9a24-4941-a792-5d70a3bb3566" (UID: "907fc1f1-9a24-4941-a792-5d70a3bb3566"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:33:37.845179 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:37.845141    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/907fc1f1-9a24-4941-a792-5d70a3bb3566-kube-api-access-qg75k" pod "907fc1f1-9a24-4941-a792-5d70a3bb3566" (UID: "907fc1f1-9a24-4941-a792-5d70a3bb3566"). InnerVolumeSpecName "kube-api-access-qg75k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:33:37.934847 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:37.934764    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/907fc1f1-9a24-4941-a792-5d70a3bb3566-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:37.934847 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:37.934799    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/907fc1f1-9a24-4941-a792-5d70a3bb3566-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:37.934847 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:37.934807    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/907fc1f1-9a24-4941-a792-5d70a3bb3566-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:37.934847 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:37.934816    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg75k\" (UniqueName: \"kubernetes.io/projected/907fc1f1-9a24-4941-a792-5d70a3bb3566-kube-api-access-qg75k\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:38.357594 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:38.357565    5012 generic.go:356] "Generic (PLEG): container finished" podID="907fc1f1-9a24-4941-a792-5d70a3bb3566" containerID="109647a616b4738d2f4051e01835ea8c4a1073ccd8f5ac33d00de9404d56b195" exitCode=0
Nov 03 00:33:38.357793 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:38.357624    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v"
Nov 03 00:33:38.357793 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:38.357643    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v" event={"ID":"907fc1f1-9a24-4941-a792-5d70a3bb3566","Type":"ContainerDied","Data":"109647a616b4738d2f4051e01835ea8c4a1073ccd8f5ac33d00de9404d56b195"}
Nov 03 00:33:38.357793 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:38.357702    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v" event={"ID":"907fc1f1-9a24-4941-a792-5d70a3bb3566","Type":"ContainerDied","Data":"40fd600035a3c77b04aba9d326c7634c6eb364d0cb51de396f735e3ecb8d47df"}
Nov 03 00:33:38.357793 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:38.357727    5012 scope.go:122] "RemoveContainer" containerID="109647a616b4738d2f4051e01835ea8c4a1073ccd8f5ac33d00de9404d56b195"
Nov 03 00:33:38.368774 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:38.368757    5012 scope.go:122] "RemoveContainer" containerID="109647a616b4738d2f4051e01835ea8c4a1073ccd8f5ac33d00de9404d56b195"
Nov 03 00:33:38.369170 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:33:38.369105    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"109647a616b4738d2f4051e01835ea8c4a1073ccd8f5ac33d00de9404d56b195\": container with ID starting with 109647a616b4738d2f4051e01835ea8c4a1073ccd8f5ac33d00de9404d56b195 not found: ID does not exist" containerID="109647a616b4738d2f4051e01835ea8c4a1073ccd8f5ac33d00de9404d56b195"
Nov 03 00:33:38.369170 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:38.369139    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"109647a616b4738d2f4051e01835ea8c4a1073ccd8f5ac33d00de9404d56b195"} err="failed to get container status \"109647a616b4738d2f4051e01835ea8c4a1073ccd8f5ac33d00de9404d56b195\": rpc error: code = NotFound desc = could not find container \"109647a616b4738d2f4051e01835ea8c4a1073ccd8f5ac33d00de9404d56b195\": container with ID starting with 109647a616b4738d2f4051e01835ea8c4a1073ccd8f5ac33d00de9404d56b195 not found: ID does not exist"
Nov 03 00:33:38.407817 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:38.407789    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v"]
Nov 03 00:33:38.417884 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:38.417861    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-191362-8579d8b649-59d6v"]
Nov 03 00:33:39.245626 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:39.245584    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="907fc1f1-9a24-4941-a792-5d70a3bb3566" path="/var/lib/kubelet/pods/907fc1f1-9a24-4941-a792-5d70a3bb3566/volumes"
Nov 03 00:33:39.683297 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:39.683266    5012 patch_prober.go:29] interesting pod/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 03 00:33:39.683757 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:39.683336    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5" podUID="14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 03 00:33:40.375893 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:40.375858    5012 generic.go:356] "Generic (PLEG): container finished" podID="14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1" containerID="0ddf9e2e137defc5edcc56e8d54690b8a3da2d0491340c0c8d72a41fba6c4332" exitCode=0
Nov 03 00:33:40.376113 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:40.375932    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5" event={"ID":"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1","Type":"ContainerDied","Data":"0ddf9e2e137defc5edcc56e8d54690b8a3da2d0491340c0c8d72a41fba6c4332"}
Nov 03 00:33:40.376308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:40.376293    5012 scope.go:122] "RemoveContainer" containerID="0ddf9e2e137defc5edcc56e8d54690b8a3da2d0491340c0c8d72a41fba6c4332"
Nov 03 00:33:40.377638 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:40.377612    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:40.377689 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:40.377679    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:40.377725 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:40.377710    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:40.823664 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:40.823580    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"]
Nov 03 00:33:40.824175 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:40.823775    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b" podUID="5138f419-8f8a-4402-b134-8fdc698d5de5" containerName="istio-proxy" containerID="cri-o://0b3e7ca758f193346e814a881ec8468b42a11c95ad52039c0de120c7234e89a5" gracePeriod=30
Nov 03 00:33:41.066070 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:41.066032    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c247619c-mb256"]
Nov 03 00:33:41.069301 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:41.069266    5012 cpu_manager.go:414] "containerMap: removing container" podUID="907fc1f1-9a24-4941-a792-5d70a3bb3566" containerName="maas-api"
Nov 03 00:33:41.069301 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:41.069297    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="907fc1f1-9a24-4941-a792-5d70a3bb3566" containerName="maas-api"
Nov 03 00:33:41.069534 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:41.069332    5012 cpu_manager.go:414] "containerMap: removing container" podUID="59529c4e-6121-4fcf-94c2-4671ab16d5e2" containerName="istio-proxy"
Nov 03 00:33:41.069534 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:41.069341    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59529c4e-6121-4fcf-94c2-4671ab16d5e2" containerName="istio-proxy"
Nov 03 00:33:41.069534 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:41.069529    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="907fc1f1-9a24-4941-a792-5d70a3bb3566" containerName="maas-api"
Nov 03 00:33:41.073637 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:41.073609    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c247619c-mb256"
Nov 03 00:33:41.077240 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:41.077173    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c247619c-mb256"]
Nov 03 00:33:41.118372 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:41.118216    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:33:41.118634 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:41.118428    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:33:41.210934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:41.210826    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad3b283d-6e55-415e-b8b0-3210e403ab1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c247619c-mb256\" (UID: \"ad3b283d-6e55-415e-b8b0-3210e403ab1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c247619c-mb256"
Nov 03 00:33:41.210934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:41.210892    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad3b283d-6e55-415e-b8b0-3210e403ab1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c247619c-mb256\" (UID: \"ad3b283d-6e55-415e-b8b0-3210e403ab1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c247619c-mb256"
Nov 03 00:33:41.211222 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:41.211187    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckg5g\" (UniqueName: \"kubernetes.io/projected/ad3b283d-6e55-415e-b8b0-3210e403ab1a-kube-api-access-ckg5g\") pod \"maas-api-revoke-sub-c247619c-mb256\" (UID: \"ad3b283d-6e55-415e-b8b0-3210e403ab1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c247619c-mb256"
Nov 03 00:33:41.312922 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:41.312857    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckg5g\" (UniqueName: \"kubernetes.io/projected/ad3b283d-6e55-415e-b8b0-3210e403ab1a-kube-api-access-ckg5g\") pod \"maas-api-revoke-sub-c247619c-mb256\" (UID: \"ad3b283d-6e55-415e-b8b0-3210e403ab1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c247619c-mb256"
Nov 03 00:33:41.313132 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:41.313112    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad3b283d-6e55-415e-b8b0-3210e403ab1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c247619c-mb256\" (UID: \"ad3b283d-6e55-415e-b8b0-3210e403ab1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c247619c-mb256"
Nov 03 00:33:41.313192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:41.313157    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad3b283d-6e55-415e-b8b0-3210e403ab1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c247619c-mb256\" (UID: \"ad3b283d-6e55-415e-b8b0-3210e403ab1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c247619c-mb256"
Nov 03 00:33:41.313816 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:41.313787    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad3b283d-6e55-415e-b8b0-3210e403ab1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c247619c-mb256\" (UID: \"ad3b283d-6e55-415e-b8b0-3210e403ab1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c247619c-mb256"
Nov 03 00:33:41.315559 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:41.315537    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad3b283d-6e55-415e-b8b0-3210e403ab1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c247619c-mb256\" (UID: \"ad3b283d-6e55-415e-b8b0-3210e403ab1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c247619c-mb256"
Nov 03 00:33:41.330597 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:41.330530    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckg5g\" (UniqueName: \"kubernetes.io/projected/ad3b283d-6e55-415e-b8b0-3210e403ab1a-kube-api-access-ckg5g\") pod \"maas-api-revoke-sub-c247619c-mb256\" (UID: \"ad3b283d-6e55-415e-b8b0-3210e403ab1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c247619c-mb256"
Nov 03 00:33:41.408935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:41.396283    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5" event={"ID":"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1","Type":"ContainerStarted","Data":"d0977a786ea130ec6b0500906d0317de15430c37be67590cf99608ab11d7de1f"}
Nov 03 00:33:41.426569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:41.426534    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c247619c-mb256"
Nov 03 00:33:41.711283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:41.711243    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:33:41.711626 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:41.711605    5012 patch_prober.go:29] interesting pod/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 03 00:33:41.711738 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:41.711647    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5" podUID="14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 03 00:33:41.781801 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:33:41.781771    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3b283d_6e55_415e_b8b0_3210e403ab1a.slice/crio-e689d8aba97f23790933dec2c25514e190f53646c8801167e909a9cea2f84745 WatchSource:0}: Error finding container e689d8aba97f23790933dec2c25514e190f53646c8801167e909a9cea2f84745: Status 404 returned error can't find the container with id e689d8aba97f23790933dec2c25514e190f53646c8801167e909a9cea2f84745
Nov 03 00:33:41.783010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:41.782975    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c247619c-mb256"]
Nov 03 00:33:42.418308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:42.418269    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c247619c-mb256" event={"ID":"ad3b283d-6e55-415e-b8b0-3210e403ab1a","Type":"ContainerStarted","Data":"9ed8649ce341881a66980839b1b144a8ad92a59e852e690745306b13c95825e1"}
Nov 03 00:33:42.418697 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:42.418325    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c247619c-mb256" event={"ID":"ad3b283d-6e55-415e-b8b0-3210e403ab1a","Type":"ContainerStarted","Data":"e689d8aba97f23790933dec2c25514e190f53646c8801167e909a9cea2f84745"}
Nov 03 00:33:42.462599 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:42.462539    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c247619c-mb256" podStartSLOduration=1.462520226 podStartE2EDuration="1.462520226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:42.453334213 +0000 UTC m=+1705.332918256" watchObservedRunningTime="2026-11-03 00:33:42.462520226 +0000 UTC m=+1705.342104270"
Nov 03 00:33:42.699642 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:42.696308    5012 patch_prober.go:29] interesting pod/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 03 00:33:42.699642 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:42.696368    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5" podUID="14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 03 00:33:43.428755 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:43.428709    5012 generic.go:356] "Generic (PLEG): container finished" podID="ad3b283d-6e55-415e-b8b0-3210e403ab1a" containerID="9ed8649ce341881a66980839b1b144a8ad92a59e852e690745306b13c95825e1" exitCode=0
Nov 03 00:33:43.428755 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:43.428749    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c247619c-mb256" event={"ID":"ad3b283d-6e55-415e-b8b0-3210e403ab1a","Type":"ContainerDied","Data":"9ed8649ce341881a66980839b1b144a8ad92a59e852e690745306b13c95825e1"}
Nov 03 00:33:43.703989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:43.696298    5012 patch_prober.go:29] interesting pod/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 03 00:33:43.703989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:43.696359    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5" podUID="14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 03 00:33:44.027151 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:44.027074    5012 patch_prober.go:29] interesting pod/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:33:44.027151 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:44.027130    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b" podUID="5138f419-8f8a-4402-b134-8fdc698d5de5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:33:44.639617 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:44.639593    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c247619c-mb256"
Nov 03 00:33:44.711004 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:44.706321    5012 patch_prober.go:29] interesting pod/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 03 00:33:44.711004 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:44.706375    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5" podUID="14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 03 00:33:44.760379 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:44.760351    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad3b283d-6e55-415e-b8b0-3210e403ab1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad3b283d-6e55-415e-b8b0-3210e403ab1a-maas-api-xks-ca\") pod \"ad3b283d-6e55-415e-b8b0-3210e403ab1a\" (UID: \"ad3b283d-6e55-415e-b8b0-3210e403ab1a\") "
Nov 03 00:33:44.760540 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:44.760528    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad3b283d-6e55-415e-b8b0-3210e403ab1a-kube-api-access-ckg5g\" (UniqueName: \"kubernetes.io/projected/ad3b283d-6e55-415e-b8b0-3210e403ab1a-kube-api-access-ckg5g\") pod \"ad3b283d-6e55-415e-b8b0-3210e403ab1a\" (UID: \"ad3b283d-6e55-415e-b8b0-3210e403ab1a\") "
Nov 03 00:33:44.760585 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:44.760552    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad3b283d-6e55-415e-b8b0-3210e403ab1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad3b283d-6e55-415e-b8b0-3210e403ab1a-maas-api-openshift-ca\") pod \"ad3b283d-6e55-415e-b8b0-3210e403ab1a\" (UID: \"ad3b283d-6e55-415e-b8b0-3210e403ab1a\") "
Nov 03 00:33:44.760985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:44.760962    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad3b283d-6e55-415e-b8b0-3210e403ab1a-maas-api-openshift-ca" pod "ad3b283d-6e55-415e-b8b0-3210e403ab1a" (UID: "ad3b283d-6e55-415e-b8b0-3210e403ab1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:33:44.762461 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:44.762423    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad3b283d-6e55-415e-b8b0-3210e403ab1a-maas-api-xks-ca" pod "ad3b283d-6e55-415e-b8b0-3210e403ab1a" (UID: "ad3b283d-6e55-415e-b8b0-3210e403ab1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:33:44.762653 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:44.762633    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3b283d-6e55-415e-b8b0-3210e403ab1a-kube-api-access-ckg5g" pod "ad3b283d-6e55-415e-b8b0-3210e403ab1a" (UID: "ad3b283d-6e55-415e-b8b0-3210e403ab1a"). InnerVolumeSpecName "kube-api-access-ckg5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:33:44.861780 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:44.861750    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad3b283d-6e55-415e-b8b0-3210e403ab1a-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:44.861780 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:44.861777    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckg5g\" (UniqueName: \"kubernetes.io/projected/ad3b283d-6e55-415e-b8b0-3210e403ab1a-kube-api-access-ckg5g\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:44.861977 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:44.861786    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad3b283d-6e55-415e-b8b0-3210e403ab1a-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:45.448317 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:45.448288    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c247619c-mb256"
Nov 03 00:33:45.448317 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:45.448299    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c247619c-mb256" event={"ID":"ad3b283d-6e55-415e-b8b0-3210e403ab1a","Type":"ContainerDied","Data":"e689d8aba97f23790933dec2c25514e190f53646c8801167e909a9cea2f84745"}
Nov 03 00:33:45.448522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:45.448344    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e689d8aba97f23790933dec2c25514e190f53646c8801167e909a9cea2f84745"
Nov 03 00:33:45.700950 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:45.699304    5012 patch_prober.go:29] interesting pod/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 03 00:33:45.700950 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:45.699375    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5" podUID="14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 03 00:33:46.226820 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.226794    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:33:46.381441 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.381404    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5138f419-8f8a-4402-b134-8fdc698d5de5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5138f419-8f8a-4402-b134-8fdc698d5de5-istiod-ca-cert\") pod \"5138f419-8f8a-4402-b134-8fdc698d5de5\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") "
Nov 03 00:33:46.381624 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.381552    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5138f419-8f8a-4402-b134-8fdc698d5de5-kube-api-access-lts5r\" (UniqueName: \"kubernetes.io/projected/5138f419-8f8a-4402-b134-8fdc698d5de5-kube-api-access-lts5r\") pod \"5138f419-8f8a-4402-b134-8fdc698d5de5\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") "
Nov 03 00:33:46.381669 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.381619    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-credential-socket\") pod \"5138f419-8f8a-4402-b134-8fdc698d5de5\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") "
Nov 03 00:33:46.381705 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.381675    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-token\" (UniqueName: \"kubernetes.io/projected/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-token\") pod \"5138f419-8f8a-4402-b134-8fdc698d5de5\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") "
Nov 03 00:33:46.381749 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.381721    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-podinfo\") pod \"5138f419-8f8a-4402-b134-8fdc698d5de5\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") "
Nov 03 00:33:46.381810 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.381775    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-data\") pod \"5138f419-8f8a-4402-b134-8fdc698d5de5\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") "
Nov 03 00:33:46.381870 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.381798    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5138f419-8f8a-4402-b134-8fdc698d5de5-istiod-ca-cert" pod "5138f419-8f8a-4402-b134-8fdc698d5de5" (UID: "5138f419-8f8a-4402-b134-8fdc698d5de5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:33:46.381870 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.381863    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-envoy\") pod \"5138f419-8f8a-4402-b134-8fdc698d5de5\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") "
Nov 03 00:33:46.382002 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.381982    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-workload-certs\") pod \"5138f419-8f8a-4402-b134-8fdc698d5de5\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") "
Nov 03 00:33:46.382073 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.382030    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-workload-socket\") pod \"5138f419-8f8a-4402-b134-8fdc698d5de5\" (UID: \"5138f419-8f8a-4402-b134-8fdc698d5de5\") "
Nov 03 00:33:46.382213 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.382189    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-data" pod "5138f419-8f8a-4402-b134-8fdc698d5de5" (UID: "5138f419-8f8a-4402-b134-8fdc698d5de5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:33:46.382507 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.382465    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-workload-socket" pod "5138f419-8f8a-4402-b134-8fdc698d5de5" (UID: "5138f419-8f8a-4402-b134-8fdc698d5de5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:33:46.382683 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.382519    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-credential-socket" pod "5138f419-8f8a-4402-b134-8fdc698d5de5" (UID: "5138f419-8f8a-4402-b134-8fdc698d5de5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:33:46.382683 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.382642    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:46.382853 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.382707    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5138f419-8f8a-4402-b134-8fdc698d5de5-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:46.382853 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.382698    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-workload-certs" pod "5138f419-8f8a-4402-b134-8fdc698d5de5" (UID: "5138f419-8f8a-4402-b134-8fdc698d5de5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:33:46.384322 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.384297    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5138f419-8f8a-4402-b134-8fdc698d5de5-kube-api-access-lts5r" pod "5138f419-8f8a-4402-b134-8fdc698d5de5" (UID: "5138f419-8f8a-4402-b134-8fdc698d5de5"). InnerVolumeSpecName "kube-api-access-lts5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:33:46.384509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.384486    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-token" pod "5138f419-8f8a-4402-b134-8fdc698d5de5" (UID: "5138f419-8f8a-4402-b134-8fdc698d5de5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:33:46.384599 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.384484    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-envoy" pod "5138f419-8f8a-4402-b134-8fdc698d5de5" (UID: "5138f419-8f8a-4402-b134-8fdc698d5de5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:33:46.384599 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.384524    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-podinfo" pod "5138f419-8f8a-4402-b134-8fdc698d5de5" (UID: "5138f419-8f8a-4402-b134-8fdc698d5de5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:33:46.458543 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.458511    5012 generic.go:356] "Generic (PLEG): container finished" podID="5138f419-8f8a-4402-b134-8fdc698d5de5" containerID="0b3e7ca758f193346e814a881ec8468b42a11c95ad52039c0de120c7234e89a5" exitCode=0
Nov 03 00:33:46.458730 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.458583    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"
Nov 03 00:33:46.458730 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.458597    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b" event={"ID":"5138f419-8f8a-4402-b134-8fdc698d5de5","Type":"ContainerDied","Data":"0b3e7ca758f193346e814a881ec8468b42a11c95ad52039c0de120c7234e89a5"}
Nov 03 00:33:46.458730 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.458667    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b" event={"ID":"5138f419-8f8a-4402-b134-8fdc698d5de5","Type":"ContainerDied","Data":"79dfd057f40fa8f0776af1d456e82134dc40f9946ffbb2d359d96f05a702ccf0"}
Nov 03 00:33:46.458730 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.458695    5012 scope.go:122] "RemoveContainer" containerID="0b3e7ca758f193346e814a881ec8468b42a11c95ad52039c0de120c7234e89a5"
Nov 03 00:33:46.473045 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.473023    5012 scope.go:122] "RemoveContainer" containerID="2e35845ac19b31c9a771449c8d3bacefb3048e0579961b3e09315714544715cc"
Nov 03 00:33:46.483018 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.482973    5012 scope.go:122] "RemoveContainer" containerID="0b3e7ca758f193346e814a881ec8468b42a11c95ad52039c0de120c7234e89a5"
Nov 03 00:33:46.483351 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:33:46.483327    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b3e7ca758f193346e814a881ec8468b42a11c95ad52039c0de120c7234e89a5\": container with ID starting with 0b3e7ca758f193346e814a881ec8468b42a11c95ad52039c0de120c7234e89a5 not found: ID does not exist" containerID="0b3e7ca758f193346e814a881ec8468b42a11c95ad52039c0de120c7234e89a5"
Nov 03 00:33:46.483459 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.483358    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b3e7ca758f193346e814a881ec8468b42a11c95ad52039c0de120c7234e89a5"} err="failed to get container status \"0b3e7ca758f193346e814a881ec8468b42a11c95ad52039c0de120c7234e89a5\": rpc error: code = NotFound desc = could not find container \"0b3e7ca758f193346e814a881ec8468b42a11c95ad52039c0de120c7234e89a5\": container with ID starting with 0b3e7ca758f193346e814a881ec8468b42a11c95ad52039c0de120c7234e89a5 not found: ID does not exist"
Nov 03 00:33:46.483459 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.483376    5012 scope.go:122] "RemoveContainer" containerID="2e35845ac19b31c9a771449c8d3bacefb3048e0579961b3e09315714544715cc"
Nov 03 00:33:46.483815 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.483791    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:46.483982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.483819    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:46.483982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.483829    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:46.483982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.483838    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lts5r\" (UniqueName: \"kubernetes.io/projected/5138f419-8f8a-4402-b134-8fdc698d5de5-kube-api-access-lts5r\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:46.483982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.483850    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5138f419-8f8a-4402-b134-8fdc698d5de5-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:46.483982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.483864    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:46.483982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.483878    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5138f419-8f8a-4402-b134-8fdc698d5de5-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:33:46.483982 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:33:46.483897    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e35845ac19b31c9a771449c8d3bacefb3048e0579961b3e09315714544715cc\": container with ID starting with 2e35845ac19b31c9a771449c8d3bacefb3048e0579961b3e09315714544715cc not found: ID does not exist" containerID="2e35845ac19b31c9a771449c8d3bacefb3048e0579961b3e09315714544715cc"
Nov 03 00:33:46.483982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.483943    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e35845ac19b31c9a771449c8d3bacefb3048e0579961b3e09315714544715cc"} err="failed to get container status \"2e35845ac19b31c9a771449c8d3bacefb3048e0579961b3e09315714544715cc\": rpc error: code = NotFound desc = could not find container \"2e35845ac19b31c9a771449c8d3bacefb3048e0579961b3e09315714544715cc\": container with ID starting with 2e35845ac19b31c9a771449c8d3bacefb3048e0579961b3e09315714544715cc not found: ID does not exist"
Nov 03 00:33:46.511041 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.511015    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"]
Nov 03 00:33:46.515447 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.515422    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-191362-openshift-default-5b645bb4fc-2tw7b"]
Nov 03 00:33:46.696382 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.696303    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:33:46.696630 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.696611    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:33:46.697246 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:46.697228    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:33:47.244996 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:47.244954    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5138f419-8f8a-4402-b134-8fdc698d5de5" path="/var/lib/kubelet/pods/5138f419-8f8a-4402-b134-8fdc698d5de5/volumes"
Nov 03 00:33:49.408127 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:49.408092    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw"]
Nov 03 00:33:49.408744 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:49.408724    5012 cpu_manager.go:414] "containerMap: removing container" podUID="5138f419-8f8a-4402-b134-8fdc698d5de5" containerName="istio-proxy"
Nov 03 00:33:49.408829 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:49.408747    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5138f419-8f8a-4402-b134-8fdc698d5de5" containerName="istio-proxy"
Nov 03 00:33:49.408829 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:49.408769    5012 cpu_manager.go:414] "containerMap: removing container" podUID="ad3b283d-6e55-415e-b8b0-3210e403ab1a" containerName="revoke-keys"
Nov 03 00:33:49.408829 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:49.408777    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3b283d-6e55-415e-b8b0-3210e403ab1a" containerName="revoke-keys"
Nov 03 00:33:49.409053 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:49.409040    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="5138f419-8f8a-4402-b134-8fdc698d5de5" containerName="istio-proxy"
Nov 03 00:33:49.409101 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:49.409056    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="5138f419-8f8a-4402-b134-8fdc698d5de5" containerName="istio-proxy"
Nov 03 00:33:49.409101 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:49.409062    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3b283d-6e55-415e-b8b0-3210e403ab1a" containerName="revoke-keys"
Nov 03 00:33:49.413799 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:49.413783    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw"
Nov 03 00:33:49.437838 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:49.437811    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-a19e67-serving-cert\""
Nov 03 00:33:49.442512 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:49.442487    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw"]
Nov 03 00:33:49.524671 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:49.524640    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c980a86-f763-4b5a-90ce-6961c1d4e947-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw\" (UID: \"0c980a86-f763-4b5a-90ce-6961c1d4e947\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw"
Nov 03 00:33:49.524810 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:49.524775    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c980a86-f763-4b5a-90ce-6961c1d4e947-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw\" (UID: \"0c980a86-f763-4b5a-90ce-6961c1d4e947\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw"
Nov 03 00:33:49.524848 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:49.524804    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c980a86-f763-4b5a-90ce-6961c1d4e947-metrics-certs\") pod \"maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw\" (UID: \"0c980a86-f763-4b5a-90ce-6961c1d4e947\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw"
Nov 03 00:33:49.524900 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:49.524873    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75gw5\" (UniqueName: \"kubernetes.io/projected/0c980a86-f763-4b5a-90ce-6961c1d4e947-kube-api-access-75gw5\") pod \"maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw\" (UID: \"0c980a86-f763-4b5a-90ce-6961c1d4e947\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw"
Nov 03 00:33:49.626201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:49.626159    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c980a86-f763-4b5a-90ce-6961c1d4e947-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw\" (UID: \"0c980a86-f763-4b5a-90ce-6961c1d4e947\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw"
Nov 03 00:33:49.626404 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:49.626303    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c980a86-f763-4b5a-90ce-6961c1d4e947-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw\" (UID: \"0c980a86-f763-4b5a-90ce-6961c1d4e947\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw"
Nov 03 00:33:49.626404 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:49.626332    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c980a86-f763-4b5a-90ce-6961c1d4e947-metrics-certs\") pod \"maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw\" (UID: \"0c980a86-f763-4b5a-90ce-6961c1d4e947\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw"
Nov 03 00:33:49.626404 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:49.626378    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75gw5\" (UniqueName: \"kubernetes.io/projected/0c980a86-f763-4b5a-90ce-6961c1d4e947-kube-api-access-75gw5\") pod \"maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw\" (UID: \"0c980a86-f763-4b5a-90ce-6961c1d4e947\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw"
Nov 03 00:33:49.627150 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:49.627119    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c980a86-f763-4b5a-90ce-6961c1d4e947-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw\" (UID: \"0c980a86-f763-4b5a-90ce-6961c1d4e947\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw"
Nov 03 00:33:49.629494 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:49.629470    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c980a86-f763-4b5a-90ce-6961c1d4e947-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw\" (UID: \"0c980a86-f763-4b5a-90ce-6961c1d4e947\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw"
Nov 03 00:33:49.629579 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:49.629470    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c980a86-f763-4b5a-90ce-6961c1d4e947-metrics-certs\") pod \"maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw\" (UID: \"0c980a86-f763-4b5a-90ce-6961c1d4e947\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw"
Nov 03 00:33:49.644318 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:49.644292    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75gw5\" (UniqueName: \"kubernetes.io/projected/0c980a86-f763-4b5a-90ce-6961c1d4e947-kube-api-access-75gw5\") pod \"maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw\" (UID: \"0c980a86-f763-4b5a-90ce-6961c1d4e947\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw"
Nov 03 00:33:49.727380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:49.727304    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw"
Nov 03 00:33:50.035857 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.035825    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw"]
Nov 03 00:33:50.036593 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:33:50.036557    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c980a86_f763_4b5a_90ce_6961c1d4e947.slice/crio-de75e38c89eb0442be6739fbdb2627d05e7666f6103116b6fc84ac3a1cd43118 WatchSource:0}: Error finding container de75e38c89eb0442be6739fbdb2627d05e7666f6103116b6fc84ac3a1cd43118: Status 404 returned error can't find the container with id de75e38c89eb0442be6739fbdb2627d05e7666f6103116b6fc84ac3a1cd43118
Nov 03 00:33:50.038240 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.038220    5012 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:33:50.493220 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.493181    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw" event={"ID":"0c980a86-f763-4b5a-90ce-6961c1d4e947","Type":"ContainerStarted","Data":"8d99d90a95c769a6624e5cb96206512c2077f85237175f6a027ad0c1d4a80a33"}
Nov 03 00:33:50.493734 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.493246    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw" event={"ID":"0c980a86-f763-4b5a-90ce-6961c1d4e947","Type":"ContainerStarted","Data":"de75e38c89eb0442be6739fbdb2627d05e7666f6103116b6fc84ac3a1cd43118"}
Nov 03 00:33:50.493734 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.493397    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw"
Nov 03 00:33:50.547933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.547579    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw" podStartSLOduration=1.252573 podStartE2EDuration="1.547557322s" totalImagesPullingTime="294.984322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:50.544205327 +0000 UTC m=+1713.423789371" watchObservedRunningTime="2026-11-03 00:33:50.547557322 +0000 UTC m=+1713.427141366"
Nov 03 00:33:50.726714 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.726675    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"]
Nov 03 00:33:50.727354 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.727340    5012 cpu_manager.go:414] "containerMap: removing container" podUID="5138f419-8f8a-4402-b134-8fdc698d5de5" containerName="istio-proxy"
Nov 03 00:33:50.727410 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.727356    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5138f419-8f8a-4402-b134-8fdc698d5de5" containerName="istio-proxy"
Nov 03 00:33:50.731268 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.731245    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.752849 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.752821    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-fc9316-openshift-default-dockercfg-8zk4q\""
Nov 03 00:33:50.758888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.758849    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"]
Nov 03 00:33:50.765956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.765898    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"]
Nov 03 00:33:50.766072 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.766010    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.771714 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.771509    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:50.775043 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.774089    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"]
Nov 03 00:33:50.775996 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.775854    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.777034 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.777011    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s9mh\" (UniqueName: \"kubernetes.io/projected/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-kube-api-access-2s9mh\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.777354 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.777324    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.777451 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.777405    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.777451 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.777438    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.777541 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.777473    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.777541 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.777532    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.777627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.777612    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.777714 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.777699    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.792378 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.792345    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"]
Nov 03 00:33:50.799876 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.799848    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"]
Nov 03 00:33:50.800262 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.800226    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.833125 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.833090    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"]
Nov 03 00:33:50.842470 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.840570    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"]
Nov 03 00:33:50.878470 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.878429    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d9034f1-0b7c-4473-93be-4d859fb850a0-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.878677 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.878485    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:50.878677 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.878562    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.878677 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.878608    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.878874 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.878695    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.878874 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.878720    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.878874 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.878754    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.878874 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.878859    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:50.879165 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.878884    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.879165 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.878920    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.879165 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.878953    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:50.879165 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.879022    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.879165 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.879046    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.879165 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.879075    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.879165 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.879114    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdvn5\" (UniqueName: \"kubernetes.io/projected/7d9034f1-0b7c-4473-93be-4d859fb850a0-kube-api-access-sdvn5\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.879165 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.879146    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.879613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.879172    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.879613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.879199    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42cd2\" (UniqueName: \"kubernetes.io/projected/e9016037-dc28-41df-a3ad-5672f6f5f0c0-kube-api-access-42cd2\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.879613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.879231    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.879613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.879262    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.879613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.879299    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:50.879613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.879329    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s9mh\" (UniqueName: \"kubernetes.io/projected/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-kube-api-access-2s9mh\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.879613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.879365    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.879613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.879412    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.879613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.879441    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:50.879613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.879465    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:50.879613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.879520    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.879613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.879553    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:50.879613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.879586    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.879613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.879605    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.880380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.879626    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.880380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.879652    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.880380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.879766    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.880380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.880087    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.880380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.880171    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2431e1a4-b1df-4516-a2bf-680be875a7d9-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:50.880380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.880219    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6xrk\" (UniqueName: \"kubernetes.io/projected/2431e1a4-b1df-4516-a2bf-680be875a7d9-kube-api-access-s6xrk\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:50.880380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.880249    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.880380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.880269    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.880380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.880284    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.880380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.880287    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.880380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.880377    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.885603 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.885580    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.887325 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.887300    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.897298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.897276    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s9mh\" (UniqueName: \"kubernetes.io/projected/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-kube-api-access-2s9mh\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.897690 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.897665    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:50.981131 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.981052    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.981131 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.981086    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:50.981131 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.981101    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:50.981467 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.981234    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:50.981467 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.981295    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.981467 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.981341    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2431e1a4-b1df-4516-a2bf-680be875a7d9-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:50.981467 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.981364    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6xrk\" (UniqueName: \"kubernetes.io/projected/2431e1a4-b1df-4516-a2bf-680be875a7d9-kube-api-access-s6xrk\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:50.981467 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.981413    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.981467 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.981437    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.981781 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.981494    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d9034f1-0b7c-4473-93be-4d859fb850a0-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.981781 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.981545    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:50.981781 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.981554    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:50.981781 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.981569    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.981781 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.981620    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:50.981781 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.981638    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.981781 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.981677    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.981781 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.981720    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.981781 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.981772    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:50.982298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.981796    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.982298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.981835    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:50.982298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.981861    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.982298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.981882    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.982298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.981888    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.982298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.981974    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.982298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.982020    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdvn5\" (UniqueName: \"kubernetes.io/projected/7d9034f1-0b7c-4473-93be-4d859fb850a0-kube-api-access-sdvn5\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.982298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.982062    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.982298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.982091    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.982298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.982121    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42cd2\" (UniqueName: \"kubernetes.io/projected/e9016037-dc28-41df-a3ad-5672f6f5f0c0-kube-api-access-42cd2\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.982298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.982160    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.982298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.982262    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.982298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.982301    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.983326 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.982338    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:50.983326 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.982517    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d9034f1-0b7c-4473-93be-4d859fb850a0-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.983326 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.982803    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:50.983326 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.983030    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2431e1a4-b1df-4516-a2bf-680be875a7d9-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:50.983326 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.983107    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.983619 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.983426    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.983949 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.983903    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.984081 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.984060    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.984225 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.984204    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.984496 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.984471    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:50.984790 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.984762    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.984881 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.984866    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.985002 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.984897    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:50.985306 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.985285    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.985394 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.985329    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:50.985850 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.985830    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:50.986532 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:50.986509    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:51.009551 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.009491    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:51.011679 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.011628    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:51.011826 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.011749    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdvn5\" (UniqueName: \"kubernetes.io/projected/7d9034f1-0b7c-4473-93be-4d859fb850a0-kube-api-access-sdvn5\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:51.012375 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.012345    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:51.013040 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.013013    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6xrk\" (UniqueName: \"kubernetes.io/projected/2431e1a4-b1df-4516-a2bf-680be875a7d9-kube-api-access-s6xrk\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:51.013603 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.013582    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42cd2\" (UniqueName: \"kubernetes.io/projected/e9016037-dc28-41df-a3ad-5672f6f5f0c0-kube-api-access-42cd2\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:51.047970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.047947    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:51.123582 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.123340    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:51.145605 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.145580    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:51.155618 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.155590    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:51.372039 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:33:51.372010    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42db167_69a8_4280_b3d6_3ac4c6a6a42f.slice/crio-f94d36ba05baf57229e7bbf2991144199b1ea130570c917aa598fdd928a20e04 WatchSource:0}: Error finding container f94d36ba05baf57229e7bbf2991144199b1ea130570c917aa598fdd928a20e04: Status 404 returned error can't find the container with id f94d36ba05baf57229e7bbf2991144199b1ea130570c917aa598fdd928a20e04
Nov 03 00:33:51.373578 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.373542    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"]
Nov 03 00:33:51.375258 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.375221    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:51.375360 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.375308    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:51.375427 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.375356    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:51.475806 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:33:51.475764    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9016037_dc28_41df_a3ad_5672f6f5f0c0.slice/crio-a08205f5d43179e2a67897347d67f46e240fbfc8b3a3eff1a0478a2d9a46a99f WatchSource:0}: Error finding container a08205f5d43179e2a67897347d67f46e240fbfc8b3a3eff1a0478a2d9a46a99f: Status 404 returned error can't find the container with id a08205f5d43179e2a67897347d67f46e240fbfc8b3a3eff1a0478a2d9a46a99f
Nov 03 00:33:51.483621 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.482635    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:51.483621 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.482734    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:51.483621 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.482781    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:51.486414 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.485010    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"]
Nov 03 00:33:51.505536 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.505498    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8" event={"ID":"e42db167-69a8-4280-b3d6-3ac4c6a6a42f","Type":"ContainerStarted","Data":"a3d5ebfb03f5055a0b57d6dce75f69ceb3cbe3464d4bd2eb8f781415028f0149"}
Nov 03 00:33:51.505985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.505562    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8" event={"ID":"e42db167-69a8-4280-b3d6-3ac4c6a6a42f","Type":"ContainerStarted","Data":"f94d36ba05baf57229e7bbf2991144199b1ea130570c917aa598fdd928a20e04"}
Nov 03 00:33:51.506976 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.506945    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx" event={"ID":"e9016037-dc28-41df-a3ad-5672f6f5f0c0","Type":"ContainerStarted","Data":"a08205f5d43179e2a67897347d67f46e240fbfc8b3a3eff1a0478a2d9a46a99f"}
Nov 03 00:33:51.525174 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.519944    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"]
Nov 03 00:33:51.537685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.537648    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"]
Nov 03 00:33:51.540452 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.540419    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:51.540634 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.540614    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:51.540771 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.540755    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:51.542727 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.542699    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:51.542801 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.542781    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:51.542859 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.542825    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:51.547207 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:51.546657    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8" podStartSLOduration=1.54664238 podStartE2EDuration="1.54664238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:51.546013023 +0000 UTC m=+1714.425597071" watchObservedRunningTime="2026-11-03 00:33:51.54664238 +0000 UTC m=+1714.426226429"
Nov 03 00:33:52.048785 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:52.048748    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:33:52.050079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:52.050043    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 03 00:33:52.050234 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:52.050105    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8" podUID="e42db167-69a8-4280-b3d6-3ac4c6a6a42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 03 00:33:52.558945 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:52.557679    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p" event={"ID":"7d9034f1-0b7c-4473-93be-4d859fb850a0","Type":"ContainerStarted","Data":"d13823617dc62bba559232a671af1bcb28e9af1518397d542be1a2d9e57342a3"}
Nov 03 00:33:52.558945 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:52.557754    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p" event={"ID":"7d9034f1-0b7c-4473-93be-4d859fb850a0","Type":"ContainerStarted","Data":"5706db1562b487c853ccbf8076ea911fc0a482025819ea115a1b606dbac5f580"}
Nov 03 00:33:52.579699 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:52.578722    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx" event={"ID":"e9016037-dc28-41df-a3ad-5672f6f5f0c0","Type":"ContainerStarted","Data":"da45fb8997b49d1b9550527be582e6b1ff58c732a4519053c19bd0badba2fd6a"}
Nov 03 00:33:52.596317 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:52.596280    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx" event={"ID":"2431e1a4-b1df-4516-a2bf-680be875a7d9","Type":"ContainerStarted","Data":"f01966f2449ee2cf6432e3e2285e05ba68cea6c1765bdd5a4b54236337e0563c"}
Nov 03 00:33:52.596690 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:52.596668    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx" event={"ID":"2431e1a4-b1df-4516-a2bf-680be875a7d9","Type":"ContainerStarted","Data":"c6ba0d424dc0eba415c96afc72ceb873cc98b23e9e8c98cf3b57ee502ce61f4a"}
Nov 03 00:33:52.709932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:52.697291    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p" podStartSLOduration=2.697269185 podStartE2EDuration="2.697269185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:52.62462031 +0000 UTC m=+1715.504204382" watchObservedRunningTime="2026-11-03 00:33:52.697269185 +0000 UTC m=+1715.576853229"
Nov 03 00:33:52.709932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:52.705309    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx" podStartSLOduration=2.705290751 podStartE2EDuration="2.705290751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:52.677673432 +0000 UTC m=+1715.557257479" watchObservedRunningTime="2026-11-03 00:33:52.705290751 +0000 UTC m=+1715.584874795"
Nov 03 00:33:52.946297 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:52.946199    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx" podStartSLOduration=2.9461756980000002 podStartE2EDuration="2.946175698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:52.728852703 +0000 UTC m=+1715.608436747" watchObservedRunningTime="2026-11-03 00:33:52.946175698 +0000 UTC m=+1715.825759741"
Nov 03 00:33:53.098791 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:53.098748    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 03 00:33:53.116166 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:53.116107    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8" podUID="e42db167-69a8-4280-b3d6-3ac4c6a6a42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 03 00:33:53.145034 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:53.144999    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:33:53.145440 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:53.145417    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 03 00:33:53.149009 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:53.148970    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx" podUID="e9016037-dc28-41df-a3ad-5672f6f5f0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 03 00:33:53.172206 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:53.172171    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:33:53.172471 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:53.172457    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:33:53.178937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:53.174004    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 03 00:33:53.179205 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:53.179154    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p" podUID="7d9034f1-0b7c-4473-93be-4d859fb850a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 03 00:33:53.184001 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:53.179403    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 03 00:33:53.219195 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:53.184212    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx" podUID="2431e1a4-b1df-4516-a2bf-680be875a7d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 03 00:33:53.732234 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:53.732189    5012 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:53.751183 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:53.751141    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:53.781041 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:53.780984    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 03 00:33:53.781659 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:53.781248    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Nov 03 00:33:53.911611 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:53.911562    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:54.038106 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:54.038001    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:54.049849 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:54.049805    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 03 00:33:54.050116 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:54.050090    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8" podUID="e42db167-69a8-4280-b3d6-3ac4c6a6a42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 03 00:33:54.124343 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:54.124305    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 03 00:33:54.124632 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:54.124602    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx" podUID="e9016037-dc28-41df-a3ad-5672f6f5f0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 03 00:33:54.154080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:54.154043    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 03 00:33:54.154553 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:54.154522    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p" podUID="7d9034f1-0b7c-4473-93be-4d859fb850a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 03 00:33:54.164496 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:54.163355    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 03 00:33:54.164496 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:54.163399    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx" podUID="2431e1a4-b1df-4516-a2bf-680be875a7d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 03 00:33:54.756947 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:54.754489    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:54.756947 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:54.755297    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:33:54.787964 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:54.784986    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:54.787964 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:54.785674    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:33:54.787964 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:54.786848    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:54.787964 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:54.787165    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:33:55.038717 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:55.038609    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:55.057348 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:55.049482    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:33:55.057348 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:55.050637    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:55.057348 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:55.050902    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 03 00:33:55.057348 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:55.050963    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:33:55.057348 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:55.050963    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8" podUID="e42db167-69a8-4280-b3d6-3ac4c6a6a42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 03 00:33:55.136934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:55.131203    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 03 00:33:55.136934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:55.131257    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx" podUID="e9016037-dc28-41df-a3ad-5672f6f5f0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 03 00:33:55.266517 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:55.266479    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 03 00:33:55.266802 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:55.266768    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx" podUID="2431e1a4-b1df-4516-a2bf-680be875a7d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 03 00:33:55.267038 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:55.267017    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 03 00:33:55.267170 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:55.267150    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p" podUID="7d9034f1-0b7c-4473-93be-4d859fb850a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 03 00:33:55.755463 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:55.754788    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded"
Nov 03 00:33:55.801876 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:55.801826    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Nov 03 00:33:55.803766 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:55.803734    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 03 00:33:55.960935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:55.956938    5012 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/health\": context deadline exceeded"
Nov 03 00:33:56.045991 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:56.045874    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 03 00:33:56.056736 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:56.056696    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 03 00:33:56.058750 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:56.058010    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 03 00:33:56.059410 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:56.059385    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8" podUID="e42db167-69a8-4280-b3d6-3ac4c6a6a42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 03 00:33:56.135807 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:56.135774    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 03 00:33:56.136679 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:56.136095    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx" podUID="e9016037-dc28-41df-a3ad-5672f6f5f0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 03 00:33:56.147985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:56.147943    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 03 00:33:56.148215 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:56.148189    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p" podUID="7d9034f1-0b7c-4473-93be-4d859fb850a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 03 00:33:56.172054 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:56.172021    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 03 00:33:56.172300 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:56.172264    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx" podUID="2431e1a4-b1df-4516-a2bf-680be875a7d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 03 00:33:56.458882 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:56.458848    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:33:56.596943 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:56.594411    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw"
Nov 03 00:33:56.785149 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:56.783998    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:56.806190 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:56.805990    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 03 00:33:57.058000 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:57.053048    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 03 00:33:57.073329 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:57.073294    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 03 00:33:57.073553 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:57.073528    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8" podUID="e42db167-69a8-4280-b3d6-3ac4c6a6a42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 03 00:33:57.074482 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:57.074454    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 03 00:33:57.128190 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:57.128152    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 03 00:33:57.151935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:57.147856    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx" podUID="e9016037-dc28-41df-a3ad-5672f6f5f0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 03 00:33:57.206014 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:57.147361    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 03 00:33:57.206283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:57.206252    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p" podUID="7d9034f1-0b7c-4473-93be-4d859fb850a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 03 00:33:57.206416 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:57.171014    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 03 00:33:57.206552 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:57.206532    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx" podUID="2431e1a4-b1df-4516-a2bf-680be875a7d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 03 00:33:57.590292 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:57.590242    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:57.838100 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:57.838044    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:58.006945 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:57.949307    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:33:58.061973 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:58.057870    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:33:58.318991 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:58.237728    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 03 00:33:58.318991 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:58.309137    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8" podUID="e42db167-69a8-4280-b3d6-3ac4c6a6a42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 03 00:33:58.353935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:58.262062    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 03 00:33:58.464960 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:58.461755    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 03 00:33:58.464960 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:58.461818    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx" podUID="e9016037-dc28-41df-a3ad-5672f6f5f0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 03 00:33:58.509164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:58.277533    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 03 00:33:58.509484 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:58.509451    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p" podUID="7d9034f1-0b7c-4473-93be-4d859fb850a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 03 00:33:58.509698 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:58.307751    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 03 00:33:58.513128 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:58.513097    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx" podUID="2431e1a4-b1df-4516-a2bf-680be875a7d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 03 00:33:58.939935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:58.939525    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:58.939935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:58.939881    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Nov 03 00:33:58.951932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:58.941268    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:33:58.964932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:58.960749    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:59.012634 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:59.012588    5012 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": context deadline exceeded"
Nov 03 00:33:59.097149 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:59.097113    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 03 00:33:59.101092 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:59.101035    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8" podUID="e42db167-69a8-4280-b3d6-3ac4c6a6a42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 03 00:33:59.125540 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:59.125495    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 03 00:33:59.125818 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:59.125783    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx" podUID="e9016037-dc28-41df-a3ad-5672f6f5f0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 03 00:33:59.148845 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:59.148815    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 03 00:33:59.149144 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:59.149103    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p" podUID="7d9034f1-0b7c-4473-93be-4d859fb850a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 03 00:33:59.159026 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:59.158869    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 03 00:33:59.159233 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:59.159211    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx" podUID="2431e1a4-b1df-4516-a2bf-680be875a7d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 03 00:33:59.371933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:59.353672    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 03 00:33:59.371933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:59.359410    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 03 00:33:59.421939 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:59.421737    5012 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:59.977932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:59.966072    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:59.977932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:59.966456    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded"
Nov 03 00:33:59.977932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:59.967205    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:33:59.994580 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:33:59.993005    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:00.076955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:00.069542    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 03 00:34:00.076955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:00.069601    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8" podUID="e42db167-69a8-4280-b3d6-3ac4c6a6a42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 03 00:34:00.156952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:00.148032    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 03 00:34:00.156952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:00.148093    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx" podUID="e9016037-dc28-41df-a3ad-5672f6f5f0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 03 00:34:00.168588 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:00.158016    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 03 00:34:00.168588 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:00.158066    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p" podUID="7d9034f1-0b7c-4473-93be-4d859fb850a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 03 00:34:00.168588 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:00.168346    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 03 00:34:00.168588 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:00.168390    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx" podUID="2431e1a4-b1df-4516-a2bf-680be875a7d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 03 00:34:00.189844 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:00.189344    5012 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/health\": context deadline exceeded"
Nov 03 00:34:00.348955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:00.346330    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:34:00.370929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:00.360996    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 03 00:34:00.370929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:00.361503    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:34:00.370929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:00.370108    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:34:00.370929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:00.370168    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:34:00.375974 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:00.374122    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:34:00.455163 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:00.451417    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:34:01.124684 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:01.124639    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 03 00:34:01.125225 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:01.124702    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx" podUID="e9016037-dc28-41df-a3ad-5672f6f5f0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 03 00:34:01.146135 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:01.146096    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 03 00:34:01.146283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:01.146183    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p" podUID="7d9034f1-0b7c-4473-93be-4d859fb850a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 03 00:34:01.157286 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:01.157255    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 03 00:34:01.157471 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:01.157300    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx" podUID="2431e1a4-b1df-4516-a2bf-680be875a7d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 03 00:34:01.810896 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:01.810862    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:34:01.812429 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:01.812405    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:34:01.823703 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:01.823678    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:34:02.342986 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:02.338226    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:34:02.342986 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:02.339754    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:34:02.350930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:02.345967    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:34:02.350930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:02.348372    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:34:02.350930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:02.349644    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:34:02.357751 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:02.357728    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:34:03.205933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:03.205027    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:34:03.205933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:03.205106    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx" podUID="2431e1a4-b1df-4516-a2bf-680be875a7d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:03.893426 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:03.891002    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:34:03.893426 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:03.892734    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:34:03.894009 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:03.893609    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:34:05.303961 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.303926    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4"]
Nov 03 00:34:05.304437 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.304205    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4" podUID="8503d3f7-ece8-4113-8888-dd5cfbe73073" containerName="maas-api" containerID="cri-o://6bccfb823cbee7d92f87dc3b9375fa646725117b956c9f290c5695d6ba21f69b" gracePeriod=30
Nov 03 00:34:05.472578 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.472545    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk"]
Nov 03 00:34:05.472800 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.472756    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk" podUID="6029bbe2-1d00-4960-8555-f5e6e1588bfa" containerName="payload-processing" containerID="cri-o://4b8770a7b9818c54e66a9903fa839da48607d492fe08995b79544048cc564f3a" gracePeriod=30
Nov 03 00:34:05.513490 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.512698    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4"]
Nov 03 00:34:05.513490 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.512959    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4" podUID="e2bd83dc-8985-4d62-86eb-4fb74ec20dd7" containerName="payload-pre-processing" containerID="cri-o://f034887dfdfdbf79ce737df951d092fc4280439972cf0961a449a081bbe48997" gracePeriod=30
Nov 03 00:34:05.655966 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.654667    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4"
Nov 03 00:34:05.790356 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.790322    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk"
Nov 03 00:34:05.802819 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.802790    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8503d3f7-ece8-4113-8888-dd5cfbe73073-kube-api-access-zxfnb\" (UniqueName: \"kubernetes.io/projected/8503d3f7-ece8-4113-8888-dd5cfbe73073-kube-api-access-zxfnb\") pod \"8503d3f7-ece8-4113-8888-dd5cfbe73073\" (UID: \"8503d3f7-ece8-4113-8888-dd5cfbe73073\") "
Nov 03 00:34:05.802992 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.802881    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8503d3f7-ece8-4113-8888-dd5cfbe73073-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8503d3f7-ece8-4113-8888-dd5cfbe73073-openshift-service-ca\") pod \"8503d3f7-ece8-4113-8888-dd5cfbe73073\" (UID: \"8503d3f7-ece8-4113-8888-dd5cfbe73073\") "
Nov 03 00:34:05.802992 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.802955    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8503d3f7-ece8-4113-8888-dd5cfbe73073-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8503d3f7-ece8-4113-8888-dd5cfbe73073-maas-api-tls\") pod \"8503d3f7-ece8-4113-8888-dd5cfbe73073\" (UID: \"8503d3f7-ece8-4113-8888-dd5cfbe73073\") "
Nov 03 00:34:05.803138 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.803046    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8503d3f7-ece8-4113-8888-dd5cfbe73073-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8503d3f7-ece8-4113-8888-dd5cfbe73073-metrics-certs\") pod \"8503d3f7-ece8-4113-8888-dd5cfbe73073\" (UID: \"8503d3f7-ece8-4113-8888-dd5cfbe73073\") "
Nov 03 00:34:05.803863 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.803824    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8503d3f7-ece8-4113-8888-dd5cfbe73073-openshift-service-ca" pod "8503d3f7-ece8-4113-8888-dd5cfbe73073" (UID: "8503d3f7-ece8-4113-8888-dd5cfbe73073"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:05.810021 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.809013    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8503d3f7-ece8-4113-8888-dd5cfbe73073-maas-api-tls" pod "8503d3f7-ece8-4113-8888-dd5cfbe73073" (UID: "8503d3f7-ece8-4113-8888-dd5cfbe73073"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:34:05.815273 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.815047    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8503d3f7-ece8-4113-8888-dd5cfbe73073-metrics-certs" pod "8503d3f7-ece8-4113-8888-dd5cfbe73073" (UID: "8503d3f7-ece8-4113-8888-dd5cfbe73073"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:34:05.815273 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.815138    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8503d3f7-ece8-4113-8888-dd5cfbe73073-kube-api-access-zxfnb" pod "8503d3f7-ece8-4113-8888-dd5cfbe73073" (UID: "8503d3f7-ece8-4113-8888-dd5cfbe73073"). InnerVolumeSpecName "kube-api-access-zxfnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:05.912735 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.906303    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6029bbe2-1d00-4960-8555-f5e6e1588bfa-kube-api-access-przph\" (UniqueName: \"kubernetes.io/projected/6029bbe2-1d00-4960-8555-f5e6e1588bfa-kube-api-access-przph\") pod \"6029bbe2-1d00-4960-8555-f5e6e1588bfa\" (UID: \"6029bbe2-1d00-4960-8555-f5e6e1588bfa\") "
Nov 03 00:34:05.912735 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.906474    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6029bbe2-1d00-4960-8555-f5e6e1588bfa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6029bbe2-1d00-4960-8555-f5e6e1588bfa-plugins-config-volume\") pod \"6029bbe2-1d00-4960-8555-f5e6e1588bfa\" (UID: \"6029bbe2-1d00-4960-8555-f5e6e1588bfa\") "
Nov 03 00:34:05.912735 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.907184    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxfnb\" (UniqueName: \"kubernetes.io/projected/8503d3f7-ece8-4113-8888-dd5cfbe73073-kube-api-access-zxfnb\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:05.912735 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.907210    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8503d3f7-ece8-4113-8888-dd5cfbe73073-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:05.912735 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.907223    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8503d3f7-ece8-4113-8888-dd5cfbe73073-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:05.912735 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.907238    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8503d3f7-ece8-4113-8888-dd5cfbe73073-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:05.913955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.913507    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6029bbe2-1d00-4960-8555-f5e6e1588bfa-plugins-config-volume" pod "6029bbe2-1d00-4960-8555-f5e6e1588bfa" (UID: "6029bbe2-1d00-4960-8555-f5e6e1588bfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:05.921829 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.921783    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6029bbe2-1d00-4960-8555-f5e6e1588bfa-kube-api-access-przph" pod "6029bbe2-1d00-4960-8555-f5e6e1588bfa" (UID: "6029bbe2-1d00-4960-8555-f5e6e1588bfa"). InnerVolumeSpecName "kube-api-access-przph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:05.949228 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.949194    5012 generic.go:356] "Generic (PLEG): container finished" podID="e2bd83dc-8985-4d62-86eb-4fb74ec20dd7" containerID="f034887dfdfdbf79ce737df951d092fc4280439972cf0961a449a081bbe48997" exitCode=0
Nov 03 00:34:05.949435 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.949399    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4" event={"ID":"e2bd83dc-8985-4d62-86eb-4fb74ec20dd7","Type":"ContainerDied","Data":"f034887dfdfdbf79ce737df951d092fc4280439972cf0961a449a081bbe48997"}
Nov 03 00:34:05.951425 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.951393    5012 generic.go:356] "Generic (PLEG): container finished" podID="6029bbe2-1d00-4960-8555-f5e6e1588bfa" containerID="4b8770a7b9818c54e66a9903fa839da48607d492fe08995b79544048cc564f3a" exitCode=0
Nov 03 00:34:05.951666 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.951615    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk" event={"ID":"6029bbe2-1d00-4960-8555-f5e6e1588bfa","Type":"ContainerDied","Data":"4b8770a7b9818c54e66a9903fa839da48607d492fe08995b79544048cc564f3a"}
Nov 03 00:34:05.951741 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.951668    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk" event={"ID":"6029bbe2-1d00-4960-8555-f5e6e1588bfa","Type":"ContainerDied","Data":"a3363fda0126e640363b9f383918fea1c31f84d87b80da7431bbc267f8f02108"}
Nov 03 00:34:05.951741 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.951698    5012 scope.go:122] "RemoveContainer" containerID="4b8770a7b9818c54e66a9903fa839da48607d492fe08995b79544048cc564f3a"
Nov 03 00:34:05.951872 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.951853    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk"
Nov 03 00:34:05.961477 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.960243    5012 generic.go:356] "Generic (PLEG): container finished" podID="8503d3f7-ece8-4113-8888-dd5cfbe73073" containerID="6bccfb823cbee7d92f87dc3b9375fa646725117b956c9f290c5695d6ba21f69b" exitCode=0
Nov 03 00:34:05.961477 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.960307    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4" event={"ID":"8503d3f7-ece8-4113-8888-dd5cfbe73073","Type":"ContainerDied","Data":"6bccfb823cbee7d92f87dc3b9375fa646725117b956c9f290c5695d6ba21f69b"}
Nov 03 00:34:05.961477 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.960351    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4" event={"ID":"8503d3f7-ece8-4113-8888-dd5cfbe73073","Type":"ContainerDied","Data":"6748e02b5430f983d27710cb2e62605ae17d4a026d05fcff116ee44bad8846c2"}
Nov 03 00:34:05.961477 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:05.960426    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4"
Nov 03 00:34:06.009334 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:06.008947    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-przph\" (UniqueName: \"kubernetes.io/projected/6029bbe2-1d00-4960-8555-f5e6e1588bfa-kube-api-access-przph\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:06.009334 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:06.008975    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6029bbe2-1d00-4960-8555-f5e6e1588bfa-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:06.021994 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:06.021970    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4"
Nov 03 00:34:06.022120 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:06.022093    5012 scope.go:122] "RemoveContainer" containerID="4b8770a7b9818c54e66a9903fa839da48607d492fe08995b79544048cc564f3a"
Nov 03 00:34:06.022885 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:34:06.022855    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b8770a7b9818c54e66a9903fa839da48607d492fe08995b79544048cc564f3a\": container with ID starting with 4b8770a7b9818c54e66a9903fa839da48607d492fe08995b79544048cc564f3a not found: ID does not exist" containerID="4b8770a7b9818c54e66a9903fa839da48607d492fe08995b79544048cc564f3a"
Nov 03 00:34:06.022991 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:06.022893    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b8770a7b9818c54e66a9903fa839da48607d492fe08995b79544048cc564f3a"} err="failed to get container status \"4b8770a7b9818c54e66a9903fa839da48607d492fe08995b79544048cc564f3a\": rpc error: code = NotFound desc = could not find container \"4b8770a7b9818c54e66a9903fa839da48607d492fe08995b79544048cc564f3a\": container with ID starting with 4b8770a7b9818c54e66a9903fa839da48607d492fe08995b79544048cc564f3a not found: ID does not exist"
Nov 03 00:34:06.022991 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:06.022950    5012 scope.go:122] "RemoveContainer" containerID="6bccfb823cbee7d92f87dc3b9375fa646725117b956c9f290c5695d6ba21f69b"
Nov 03 00:34:06.033534 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:06.033510    5012 scope.go:122] "RemoveContainer" containerID="6bccfb823cbee7d92f87dc3b9375fa646725117b956c9f290c5695d6ba21f69b"
Nov 03 00:34:06.033864 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:34:06.033843    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bccfb823cbee7d92f87dc3b9375fa646725117b956c9f290c5695d6ba21f69b\": container with ID starting with 6bccfb823cbee7d92f87dc3b9375fa646725117b956c9f290c5695d6ba21f69b not found: ID does not exist" containerID="6bccfb823cbee7d92f87dc3b9375fa646725117b956c9f290c5695d6ba21f69b"
Nov 03 00:34:06.033997 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:06.033877    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bccfb823cbee7d92f87dc3b9375fa646725117b956c9f290c5695d6ba21f69b"} err="failed to get container status \"6bccfb823cbee7d92f87dc3b9375fa646725117b956c9f290c5695d6ba21f69b\": rpc error: code = NotFound desc = could not find container \"6bccfb823cbee7d92f87dc3b9375fa646725117b956c9f290c5695d6ba21f69b\": container with ID starting with 6bccfb823cbee7d92f87dc3b9375fa646725117b956c9f290c5695d6ba21f69b not found: ID does not exist"
Nov 03 00:34:06.046683 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:06.046661    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk"]
Nov 03 00:34:06.050663 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:06.050640    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a43474-8587dfd448-n6vbk"]
Nov 03 00:34:06.080321 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:06.080298    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4"]
Nov 03 00:34:06.083769 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:06.083749    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a43474-6fc4c886bc-hhpc4"]
Nov 03 00:34:06.211287 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:06.211205    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2bd83dc-8985-4d62-86eb-4fb74ec20dd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2bd83dc-8985-4d62-86eb-4fb74ec20dd7-plugins-config-volume\") pod \"e2bd83dc-8985-4d62-86eb-4fb74ec20dd7\" (UID: \"e2bd83dc-8985-4d62-86eb-4fb74ec20dd7\") "
Nov 03 00:34:06.211451 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:06.211301    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2bd83dc-8985-4d62-86eb-4fb74ec20dd7-kube-api-access-plr7q\" (UniqueName: \"kubernetes.io/projected/e2bd83dc-8985-4d62-86eb-4fb74ec20dd7-kube-api-access-plr7q\") pod \"e2bd83dc-8985-4d62-86eb-4fb74ec20dd7\" (UID: \"e2bd83dc-8985-4d62-86eb-4fb74ec20dd7\") "
Nov 03 00:34:06.211583 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:06.211560    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2bd83dc-8985-4d62-86eb-4fb74ec20dd7-plugins-config-volume" pod "e2bd83dc-8985-4d62-86eb-4fb74ec20dd7" (UID: "e2bd83dc-8985-4d62-86eb-4fb74ec20dd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:06.211845 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:06.211826    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2bd83dc-8985-4d62-86eb-4fb74ec20dd7-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:06.213406 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:06.213383    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2bd83dc-8985-4d62-86eb-4fb74ec20dd7-kube-api-access-plr7q" pod "e2bd83dc-8985-4d62-86eb-4fb74ec20dd7" (UID: "e2bd83dc-8985-4d62-86eb-4fb74ec20dd7"). InnerVolumeSpecName "kube-api-access-plr7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:06.313316 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:06.313285    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plr7q\" (UniqueName: \"kubernetes.io/projected/e2bd83dc-8985-4d62-86eb-4fb74ec20dd7-kube-api-access-plr7q\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:06.971813 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:06.971785    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4"
Nov 03 00:34:06.972022 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:06.971791    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4" event={"ID":"e2bd83dc-8985-4d62-86eb-4fb74ec20dd7","Type":"ContainerDied","Data":"d3229a744ce01a4500f5935c1533682a6b5addb546332c44a44d84852bac1ff9"}
Nov 03 00:34:06.972022 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:06.971950    5012 scope.go:122] "RemoveContainer" containerID="f034887dfdfdbf79ce737df951d092fc4280439972cf0961a449a081bbe48997"
Nov 03 00:34:07.023578 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:07.023542    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4"]
Nov 03 00:34:07.033341 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:07.033310    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a43474-bf9cb9579-xt2x4"]
Nov 03 00:34:07.246126 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:07.246033    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6029bbe2-1d00-4960-8555-f5e6e1588bfa" path="/var/lib/kubelet/pods/6029bbe2-1d00-4960-8555-f5e6e1588bfa/volumes"
Nov 03 00:34:07.246611 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:07.246589    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8503d3f7-ece8-4113-8888-dd5cfbe73073" path="/var/lib/kubelet/pods/8503d3f7-ece8-4113-8888-dd5cfbe73073/volumes"
Nov 03 00:34:07.247111 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:07.247091    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2bd83dc-8985-4d62-86eb-4fb74ec20dd7" path="/var/lib/kubelet/pods/e2bd83dc-8985-4d62-86eb-4fb74ec20dd7/volumes"
Nov 03 00:34:07.365771 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:34:07.365737    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:34:08.268520 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:08.268487    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"]
Nov 03 00:34:08.268784 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:08.268759    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h" podUID="b4f51e9c-5f2c-41df-a26f-4978e47bf980" containerName="istio-proxy" containerID="cri-o://ac896fba81a2ef1ae9f6c3a3e8af8e0048916bc7ee3b6bbf9a023fba745f2765" gracePeriod=30
Nov 03 00:34:13.582079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.582055    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:34:13.621599 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.621572    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-workload-socket\") pod \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") "
Nov 03 00:34:13.621730 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.621604    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-podinfo\") pod \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") "
Nov 03 00:34:13.621730 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.621648    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-workload-certs\") pod \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") "
Nov 03 00:34:13.621730 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.621705    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4f51e9c-5f2c-41df-a26f-4978e47bf980-kube-api-access-7ss5z\" (UniqueName: \"kubernetes.io/projected/b4f51e9c-5f2c-41df-a26f-4978e47bf980-kube-api-access-7ss5z\") pod \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") "
Nov 03 00:34:13.622080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.621760    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istiod-ca-cert\") pod \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") "
Nov 03 00:34:13.622080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.621850    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-data\") pod \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") "
Nov 03 00:34:13.622080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.621879    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-envoy\") pod \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") "
Nov 03 00:34:13.622080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.621949    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-workload-socket" pod "b4f51e9c-5f2c-41df-a26f-4978e47bf980" (UID: "b4f51e9c-5f2c-41df-a26f-4978e47bf980"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:34:13.622080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.621967    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-credential-socket\") pod \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") "
Nov 03 00:34:13.622080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.622009    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-token\" (UniqueName: \"kubernetes.io/projected/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-token\") pod \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\" (UID: \"b4f51e9c-5f2c-41df-a26f-4978e47bf980\") "
Nov 03 00:34:13.622410 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.622082    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-workload-certs" pod "b4f51e9c-5f2c-41df-a26f-4978e47bf980" (UID: "b4f51e9c-5f2c-41df-a26f-4978e47bf980"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:34:13.622492 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.622419    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-data" pod "b4f51e9c-5f2c-41df-a26f-4978e47bf980" (UID: "b4f51e9c-5f2c-41df-a26f-4978e47bf980"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:34:13.622577 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.622543    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istiod-ca-cert" pod "b4f51e9c-5f2c-41df-a26f-4978e47bf980" (UID: "b4f51e9c-5f2c-41df-a26f-4978e47bf980"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:13.622814 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.622784    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-credential-socket" pod "b4f51e9c-5f2c-41df-a26f-4978e47bf980" (UID: "b4f51e9c-5f2c-41df-a26f-4978e47bf980"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:34:13.622905 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.622827    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:13.622905 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.622869    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:13.622905 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.622884    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:13.622905 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.622898    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:13.624184 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.624160    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-podinfo" pod "b4f51e9c-5f2c-41df-a26f-4978e47bf980" (UID: "b4f51e9c-5f2c-41df-a26f-4978e47bf980"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:34:13.624344 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.624322    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f51e9c-5f2c-41df-a26f-4978e47bf980-kube-api-access-7ss5z" pod "b4f51e9c-5f2c-41df-a26f-4978e47bf980" (UID: "b4f51e9c-5f2c-41df-a26f-4978e47bf980"). InnerVolumeSpecName "kube-api-access-7ss5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:13.624418 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.624346    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-token" pod "b4f51e9c-5f2c-41df-a26f-4978e47bf980" (UID: "b4f51e9c-5f2c-41df-a26f-4978e47bf980"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:13.624538 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.624523    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-envoy" pod "b4f51e9c-5f2c-41df-a26f-4978e47bf980" (UID: "b4f51e9c-5f2c-41df-a26f-4978e47bf980"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:34:13.723849 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.723815    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:13.723849 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.723845    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f51e9c-5f2c-41df-a26f-4978e47bf980-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:13.724070 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.723859    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:13.724070 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.723873    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4f51e9c-5f2c-41df-a26f-4978e47bf980-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:13.724070 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:13.723884    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ss5z\" (UniqueName: \"kubernetes.io/projected/b4f51e9c-5f2c-41df-a26f-4978e47bf980-kube-api-access-7ss5z\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:14.045015 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:14.044977    5012 generic.go:356] "Generic (PLEG): container finished" podID="b4f51e9c-5f2c-41df-a26f-4978e47bf980" containerID="ac896fba81a2ef1ae9f6c3a3e8af8e0048916bc7ee3b6bbf9a023fba745f2765" exitCode=0
Nov 03 00:34:14.045203 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:14.045033    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"
Nov 03 00:34:14.045203 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:14.045064    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h" event={"ID":"b4f51e9c-5f2c-41df-a26f-4978e47bf980","Type":"ContainerDied","Data":"ac896fba81a2ef1ae9f6c3a3e8af8e0048916bc7ee3b6bbf9a023fba745f2765"}
Nov 03 00:34:14.045203 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:14.045135    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h" event={"ID":"b4f51e9c-5f2c-41df-a26f-4978e47bf980","Type":"ContainerDied","Data":"962a8d3b70763ea9ae237214e3f30eeaf0b54491ab66b78d385d1c2d5a26f61f"}
Nov 03 00:34:14.045203 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:14.045165    5012 scope.go:122] "RemoveContainer" containerID="ac896fba81a2ef1ae9f6c3a3e8af8e0048916bc7ee3b6bbf9a023fba745f2765"
Nov 03 00:34:14.056768 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:14.056749    5012 scope.go:122] "RemoveContainer" containerID="81e9a006c55992de198eb569c21ec23b96c9a96cfc3c317c947c754e27f334d4"
Nov 03 00:34:14.072407 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:14.072383    5012 scope.go:122] "RemoveContainer" containerID="ac896fba81a2ef1ae9f6c3a3e8af8e0048916bc7ee3b6bbf9a023fba745f2765"
Nov 03 00:34:14.072865 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:34:14.072843    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac896fba81a2ef1ae9f6c3a3e8af8e0048916bc7ee3b6bbf9a023fba745f2765\": container with ID starting with ac896fba81a2ef1ae9f6c3a3e8af8e0048916bc7ee3b6bbf9a023fba745f2765 not found: ID does not exist" containerID="ac896fba81a2ef1ae9f6c3a3e8af8e0048916bc7ee3b6bbf9a023fba745f2765"
Nov 03 00:34:14.072957 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:14.072876    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac896fba81a2ef1ae9f6c3a3e8af8e0048916bc7ee3b6bbf9a023fba745f2765"} err="failed to get container status \"ac896fba81a2ef1ae9f6c3a3e8af8e0048916bc7ee3b6bbf9a023fba745f2765\": rpc error: code = NotFound desc = could not find container \"ac896fba81a2ef1ae9f6c3a3e8af8e0048916bc7ee3b6bbf9a023fba745f2765\": container with ID starting with ac896fba81a2ef1ae9f6c3a3e8af8e0048916bc7ee3b6bbf9a023fba745f2765 not found: ID does not exist"
Nov 03 00:34:14.072957 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:14.072898    5012 scope.go:122] "RemoveContainer" containerID="81e9a006c55992de198eb569c21ec23b96c9a96cfc3c317c947c754e27f334d4"
Nov 03 00:34:14.073223 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:34:14.073204    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81e9a006c55992de198eb569c21ec23b96c9a96cfc3c317c947c754e27f334d4\": container with ID starting with 81e9a006c55992de198eb569c21ec23b96c9a96cfc3c317c947c754e27f334d4 not found: ID does not exist" containerID="81e9a006c55992de198eb569c21ec23b96c9a96cfc3c317c947c754e27f334d4"
Nov 03 00:34:14.073283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:14.073229    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81e9a006c55992de198eb569c21ec23b96c9a96cfc3c317c947c754e27f334d4"} err="failed to get container status \"81e9a006c55992de198eb569c21ec23b96c9a96cfc3c317c947c754e27f334d4\": rpc error: code = NotFound desc = could not find container \"81e9a006c55992de198eb569c21ec23b96c9a96cfc3c317c947c754e27f334d4\": container with ID starting with 81e9a006c55992de198eb569c21ec23b96c9a96cfc3c317c947c754e27f334d4 not found: ID does not exist"
Nov 03 00:34:14.096213 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:14.096159    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"]
Nov 03 00:34:14.100238 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:14.100214    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a43474-openshift-default-7dccdbb857-g658h"]
Nov 03 00:34:15.245100 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:15.245071    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4f51e9c-5f2c-41df-a26f-4978e47bf980" path="/var/lib/kubelet/pods/b4f51e9c-5f2c-41df-a26f-4978e47bf980/volumes"
Nov 03 00:34:18.395937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:18.375175    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:18.523332 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:18.523290    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:34:18.535087 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:18.535032    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p" podUID="7d9034f1-0b7c-4473-93be-4d859fb850a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:18.545012 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:18.539839    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:18.581992 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:18.579000    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:34:18.582293 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:18.582252    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx" podUID="e9016037-dc28-41df-a3ad-5672f6f5f0c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:18.590177 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:18.590131    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:18.826699 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:18.825801    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 03 00:34:18.989973 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:18.989261    5012 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:19.063513 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:19.062958    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 03 00:34:19.541811 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:19.541776    5012 scope.go:122] "RemoveContainer" containerID="6a7e5da61572eacda0356ce6dc7e2df213d86b4bbc8ff2fc5642950d37f7d780"
Nov 03 00:34:28.294333 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:28.294293    5012 generic.go:356] "Generic (PLEG): container finished" podID="6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5" containerID="45a709ea2c77440cc22cf716ce9e4b2d0eeefe0c48ed89ab9faa359d04fbffd3" exitCode=0
Nov 03 00:34:28.294799 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:28.294365    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln" event={"ID":"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5","Type":"ContainerDied","Data":"45a709ea2c77440cc22cf716ce9e4b2d0eeefe0c48ed89ab9faa359d04fbffd3"}
Nov 03 00:34:28.294875 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:28.294834    5012 scope.go:122] "RemoveContainer" containerID="45a709ea2c77440cc22cf716ce9e4b2d0eeefe0c48ed89ab9faa359d04fbffd3"
Nov 03 00:34:28.296422 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:28.296380    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:34:28.296527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:28.296467    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:34:28.296527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:28.296517    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:34:29.269944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:29.269881    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr"]
Nov 03 00:34:29.270157 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:29.270093    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr" podUID="fb399f46-4a86-4660-bfef-c6267eaff189" containerName="payload-processing" containerID="cri-o://3d49eb56f198e697eae77dace597f0cc8c30f9716fbdcc1bae30472126814f0a" gracePeriod=30
Nov 03 00:34:29.336615 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:29.336580    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q"]
Nov 03 00:34:29.337258 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:29.337224    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q" podUID="ef6038ec-0566-4dd7-adb6-33ac116c4c75" containerName="payload-pre-processing" containerID="cri-o://4edb58b558d867f1001a71202ebaf481ff9d5b65896025b72e8fb35b512ec144" gracePeriod=30
Nov 03 00:34:29.340516 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:29.340480    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln" event={"ID":"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5","Type":"ContainerStarted","Data":"8d1fb92ef148cf95773de678772db9c73a61acad6dff5adf202199fd22a23974"}
Nov 03 00:34:29.840026 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:29.839985    5012 patch_prober.go:29] interesting pod/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 03 00:34:29.840226 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:29.840055    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln" podUID="6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 03 00:34:29.840388 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:29.840362    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:34:29.865353 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:29.865294    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr"
Nov 03 00:34:29.885991 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:29.884624    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb399f46-4a86-4660-bfef-c6267eaff189-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb399f46-4a86-4660-bfef-c6267eaff189-plugins-config-volume\") pod \"fb399f46-4a86-4660-bfef-c6267eaff189\" (UID: \"fb399f46-4a86-4660-bfef-c6267eaff189\") "
Nov 03 00:34:29.885991 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:29.884680    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb399f46-4a86-4660-bfef-c6267eaff189-kube-api-access-vj97k\" (UniqueName: \"kubernetes.io/projected/fb399f46-4a86-4660-bfef-c6267eaff189-kube-api-access-vj97k\") pod \"fb399f46-4a86-4660-bfef-c6267eaff189\" (UID: \"fb399f46-4a86-4660-bfef-c6267eaff189\") "
Nov 03 00:34:29.885991 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:29.885276    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb399f46-4a86-4660-bfef-c6267eaff189-plugins-config-volume" pod "fb399f46-4a86-4660-bfef-c6267eaff189" (UID: "fb399f46-4a86-4660-bfef-c6267eaff189"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:29.930592 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:29.930539    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb399f46-4a86-4660-bfef-c6267eaff189-kube-api-access-vj97k" pod "fb399f46-4a86-4660-bfef-c6267eaff189" (UID: "fb399f46-4a86-4660-bfef-c6267eaff189"). InnerVolumeSpecName "kube-api-access-vj97k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:29.991966 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:29.988049    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb399f46-4a86-4660-bfef-c6267eaff189-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:29.991966 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:29.988111    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj97k\" (UniqueName: \"kubernetes.io/projected/fb399f46-4a86-4660-bfef-c6267eaff189-kube-api-access-vj97k\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:30.014212 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.014165    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q"
Nov 03 00:34:30.119431 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.118492    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6038ec-0566-4dd7-adb6-33ac116c4c75-kube-api-access-bqbb5\" (UniqueName: \"kubernetes.io/projected/ef6038ec-0566-4dd7-adb6-33ac116c4c75-kube-api-access-bqbb5\") pod \"ef6038ec-0566-4dd7-adb6-33ac116c4c75\" (UID: \"ef6038ec-0566-4dd7-adb6-33ac116c4c75\") "
Nov 03 00:34:30.119431 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.118702    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef6038ec-0566-4dd7-adb6-33ac116c4c75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6038ec-0566-4dd7-adb6-33ac116c4c75-plugins-config-volume\") pod \"ef6038ec-0566-4dd7-adb6-33ac116c4c75\" (UID: \"ef6038ec-0566-4dd7-adb6-33ac116c4c75\") "
Nov 03 00:34:30.119431 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.119331    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef6038ec-0566-4dd7-adb6-33ac116c4c75-plugins-config-volume" pod "ef6038ec-0566-4dd7-adb6-33ac116c4c75" (UID: "ef6038ec-0566-4dd7-adb6-33ac116c4c75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:30.121369 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.121339    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6038ec-0566-4dd7-adb6-33ac116c4c75-kube-api-access-bqbb5" pod "ef6038ec-0566-4dd7-adb6-33ac116c4c75" (UID: "ef6038ec-0566-4dd7-adb6-33ac116c4c75"). InnerVolumeSpecName "kube-api-access-bqbb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:30.221301 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.221253    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6038ec-0566-4dd7-adb6-33ac116c4c75-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:30.221301 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.221301    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqbb5\" (UniqueName: \"kubernetes.io/projected/ef6038ec-0566-4dd7-adb6-33ac116c4c75-kube-api-access-bqbb5\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:30.224481 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.224454    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:34:30.351117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.351076    5012 generic.go:356] "Generic (PLEG): container finished" podID="fb399f46-4a86-4660-bfef-c6267eaff189" containerID="3d49eb56f198e697eae77dace597f0cc8c30f9716fbdcc1bae30472126814f0a" exitCode=0
Nov 03 00:34:30.351635 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.351137    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr"
Nov 03 00:34:30.351635 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.351150    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr" event={"ID":"fb399f46-4a86-4660-bfef-c6267eaff189","Type":"ContainerDied","Data":"3d49eb56f198e697eae77dace597f0cc8c30f9716fbdcc1bae30472126814f0a"}
Nov 03 00:34:30.351635 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.351209    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr" event={"ID":"fb399f46-4a86-4660-bfef-c6267eaff189","Type":"ContainerDied","Data":"77324d81d44619c139610a7480f5be4b741743bf075786ac9ddab432a83b8a2f"}
Nov 03 00:34:30.351635 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.351235    5012 scope.go:122] "RemoveContainer" containerID="3d49eb56f198e697eae77dace597f0cc8c30f9716fbdcc1bae30472126814f0a"
Nov 03 00:34:30.352838 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.352815    5012 generic.go:356] "Generic (PLEG): container finished" podID="ef6038ec-0566-4dd7-adb6-33ac116c4c75" containerID="4edb58b558d867f1001a71202ebaf481ff9d5b65896025b72e8fb35b512ec144" exitCode=0
Nov 03 00:34:30.352968 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.352859    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q"
Nov 03 00:34:30.352968 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.352902    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q" event={"ID":"ef6038ec-0566-4dd7-adb6-33ac116c4c75","Type":"ContainerDied","Data":"4edb58b558d867f1001a71202ebaf481ff9d5b65896025b72e8fb35b512ec144"}
Nov 03 00:34:30.353095 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.352998    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q" event={"ID":"ef6038ec-0566-4dd7-adb6-33ac116c4c75","Type":"ContainerDied","Data":"0cecd3f820e6e633eca69df5f19758b8dd39fa2c6363a9fd1bed9bc0184c5c48"}
Nov 03 00:34:30.364722 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.364703    5012 scope.go:122] "RemoveContainer" containerID="3d49eb56f198e697eae77dace597f0cc8c30f9716fbdcc1bae30472126814f0a"
Nov 03 00:34:30.365075 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:34:30.365055    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d49eb56f198e697eae77dace597f0cc8c30f9716fbdcc1bae30472126814f0a\": container with ID starting with 3d49eb56f198e697eae77dace597f0cc8c30f9716fbdcc1bae30472126814f0a not found: ID does not exist" containerID="3d49eb56f198e697eae77dace597f0cc8c30f9716fbdcc1bae30472126814f0a"
Nov 03 00:34:30.365128 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.365086    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d49eb56f198e697eae77dace597f0cc8c30f9716fbdcc1bae30472126814f0a"} err="failed to get container status \"3d49eb56f198e697eae77dace597f0cc8c30f9716fbdcc1bae30472126814f0a\": rpc error: code = NotFound desc = could not find container \"3d49eb56f198e697eae77dace597f0cc8c30f9716fbdcc1bae30472126814f0a\": container with ID starting with 3d49eb56f198e697eae77dace597f0cc8c30f9716fbdcc1bae30472126814f0a not found: ID does not exist"
Nov 03 00:34:30.365128 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.365103    5012 scope.go:122] "RemoveContainer" containerID="4edb58b558d867f1001a71202ebaf481ff9d5b65896025b72e8fb35b512ec144"
Nov 03 00:34:30.376101 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.376071    5012 scope.go:122] "RemoveContainer" containerID="4edb58b558d867f1001a71202ebaf481ff9d5b65896025b72e8fb35b512ec144"
Nov 03 00:34:30.376469 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:34:30.376447    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4edb58b558d867f1001a71202ebaf481ff9d5b65896025b72e8fb35b512ec144\": container with ID starting with 4edb58b558d867f1001a71202ebaf481ff9d5b65896025b72e8fb35b512ec144 not found: ID does not exist" containerID="4edb58b558d867f1001a71202ebaf481ff9d5b65896025b72e8fb35b512ec144"
Nov 03 00:34:30.376620 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.376476    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4edb58b558d867f1001a71202ebaf481ff9d5b65896025b72e8fb35b512ec144"} err="failed to get container status \"4edb58b558d867f1001a71202ebaf481ff9d5b65896025b72e8fb35b512ec144\": rpc error: code = NotFound desc = could not find container \"4edb58b558d867f1001a71202ebaf481ff9d5b65896025b72e8fb35b512ec144\": container with ID starting with 4edb58b558d867f1001a71202ebaf481ff9d5b65896025b72e8fb35b512ec144 not found: ID does not exist"
Nov 03 00:34:30.425119 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.425079    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr"]
Nov 03 00:34:30.434536 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.434507    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-6ff766ff5b-2vpdr"]
Nov 03 00:34:30.463726 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.463685    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q"]
Nov 03 00:34:30.467473 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.467445    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-549f76c54c-w575q"]
Nov 03 00:34:30.838058 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.837973    5012 patch_prober.go:29] interesting pod/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 03 00:34:30.838058 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:30.838039    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln" podUID="6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 03 00:34:31.245011 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:31.244976    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef6038ec-0566-4dd7-adb6-33ac116c4c75" path="/var/lib/kubelet/pods/ef6038ec-0566-4dd7-adb6-33ac116c4c75/volumes"
Nov 03 00:34:31.245531 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:31.245511    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb399f46-4a86-4660-bfef-c6267eaff189" path="/var/lib/kubelet/pods/fb399f46-4a86-4660-bfef-c6267eaff189/volumes"
Nov 03 00:34:31.838465 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:31.838410    5012 patch_prober.go:29] interesting pod/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 03 00:34:31.839030 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:31.838998    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln" podUID="6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 03 00:34:32.837597 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:32.837557    5012 patch_prober.go:29] interesting pod/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 03 00:34:32.837809 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:32.837618    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln" podUID="6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 03 00:34:33.838383 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:33.838344    5012 patch_prober.go:29] interesting pod/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 03 00:34:33.838811 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:33.838399    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln" podUID="6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 03 00:34:34.327970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.325756    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"]
Nov 03 00:34:34.327970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.326973    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5" podUID="14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1" containerName="istio-proxy" containerID="cri-o://d0977a786ea130ec6b0500906d0317de15430c37be67590cf99608ab11d7de1f" gracePeriod=30
Nov 03 00:34:34.380065 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.380027    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26c136dc-l88v7"]
Nov 03 00:34:34.380717 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.380696    5012 cpu_manager.go:414] "containerMap: removing container" podUID="b4f51e9c-5f2c-41df-a26f-4978e47bf980" containerName="istio-proxy"
Nov 03 00:34:34.380822 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.380718    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4f51e9c-5f2c-41df-a26f-4978e47bf980" containerName="istio-proxy"
Nov 03 00:34:34.380822 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.380735    5012 cpu_manager.go:414] "containerMap: removing container" podUID="8503d3f7-ece8-4113-8888-dd5cfbe73073" containerName="maas-api"
Nov 03 00:34:34.380822 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.380742    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8503d3f7-ece8-4113-8888-dd5cfbe73073" containerName="maas-api"
Nov 03 00:34:34.380822 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.380756    5012 cpu_manager.go:414] "containerMap: removing container" podUID="6029bbe2-1d00-4960-8555-f5e6e1588bfa" containerName="payload-processing"
Nov 03 00:34:34.380822 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.380764    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6029bbe2-1d00-4960-8555-f5e6e1588bfa" containerName="payload-processing"
Nov 03 00:34:34.380822 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.380793    5012 cpu_manager.go:414] "containerMap: removing container" podUID="b4f51e9c-5f2c-41df-a26f-4978e47bf980" containerName="istio-proxy"
Nov 03 00:34:34.380822 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.380800    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4f51e9c-5f2c-41df-a26f-4978e47bf980" containerName="istio-proxy"
Nov 03 00:34:34.380822 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.380818    5012 cpu_manager.go:414] "containerMap: removing container" podUID="e2bd83dc-8985-4d62-86eb-4fb74ec20dd7" containerName="payload-pre-processing"
Nov 03 00:34:34.381308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.380825    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2bd83dc-8985-4d62-86eb-4fb74ec20dd7" containerName="payload-pre-processing"
Nov 03 00:34:34.381308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.380840    5012 cpu_manager.go:414] "containerMap: removing container" podUID="ef6038ec-0566-4dd7-adb6-33ac116c4c75" containerName="payload-pre-processing"
Nov 03 00:34:34.381308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.380848    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6038ec-0566-4dd7-adb6-33ac116c4c75" containerName="payload-pre-processing"
Nov 03 00:34:34.381308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.380896    5012 cpu_manager.go:414] "containerMap: removing container" podUID="fb399f46-4a86-4660-bfef-c6267eaff189" containerName="payload-processing"
Nov 03 00:34:34.381308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.380904    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb399f46-4a86-4660-bfef-c6267eaff189" containerName="payload-processing"
Nov 03 00:34:34.381308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.381072    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4f51e9c-5f2c-41df-a26f-4978e47bf980" containerName="istio-proxy"
Nov 03 00:34:34.381308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.381088    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="8503d3f7-ece8-4113-8888-dd5cfbe73073" containerName="maas-api"
Nov 03 00:34:34.381308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.381099    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="6029bbe2-1d00-4960-8555-f5e6e1588bfa" containerName="payload-processing"
Nov 03 00:34:34.381308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.381112    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2bd83dc-8985-4d62-86eb-4fb74ec20dd7" containerName="payload-pre-processing"
Nov 03 00:34:34.381308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.381124    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6038ec-0566-4dd7-adb6-33ac116c4c75" containerName="payload-pre-processing"
Nov 03 00:34:34.381308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.381153    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb399f46-4a86-4660-bfef-c6267eaff189" containerName="payload-processing"
Nov 03 00:34:34.385061 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.385035    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c136dc-l88v7"
Nov 03 00:34:34.459932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.448881    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26c136dc-l88v7"]
Nov 03 00:34:34.477959 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.474342    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:34:34.483265 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.483229    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:34:34.533204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.484731    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f598db0-3077-44f7-ba00-40ab57c0a64b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26c136dc-l88v7\" (UID: \"1f598db0-3077-44f7-ba00-40ab57c0a64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c136dc-l88v7"
Nov 03 00:34:34.533757 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.533719    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw4nh\" (UniqueName: \"kubernetes.io/projected/1f598db0-3077-44f7-ba00-40ab57c0a64b-kube-api-access-gw4nh\") pod \"maas-api-revoke-sub-26c136dc-l88v7\" (UID: \"1f598db0-3077-44f7-ba00-40ab57c0a64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c136dc-l88v7"
Nov 03 00:34:34.540448 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.533996    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f598db0-3077-44f7-ba00-40ab57c0a64b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26c136dc-l88v7\" (UID: \"1f598db0-3077-44f7-ba00-40ab57c0a64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c136dc-l88v7"
Nov 03 00:34:34.641929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.641869    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw4nh\" (UniqueName: \"kubernetes.io/projected/1f598db0-3077-44f7-ba00-40ab57c0a64b-kube-api-access-gw4nh\") pod \"maas-api-revoke-sub-26c136dc-l88v7\" (UID: \"1f598db0-3077-44f7-ba00-40ab57c0a64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c136dc-l88v7"
Nov 03 00:34:34.642157 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.641985    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f598db0-3077-44f7-ba00-40ab57c0a64b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26c136dc-l88v7\" (UID: \"1f598db0-3077-44f7-ba00-40ab57c0a64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c136dc-l88v7"
Nov 03 00:34:34.642157 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.642144    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f598db0-3077-44f7-ba00-40ab57c0a64b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26c136dc-l88v7\" (UID: \"1f598db0-3077-44f7-ba00-40ab57c0a64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c136dc-l88v7"
Nov 03 00:34:34.644009 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.643854    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f598db0-3077-44f7-ba00-40ab57c0a64b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26c136dc-l88v7\" (UID: \"1f598db0-3077-44f7-ba00-40ab57c0a64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c136dc-l88v7"
Nov 03 00:34:34.645555 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.645531    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f598db0-3077-44f7-ba00-40ab57c0a64b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26c136dc-l88v7\" (UID: \"1f598db0-3077-44f7-ba00-40ab57c0a64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c136dc-l88v7"
Nov 03 00:34:34.669219 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.668897    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw4nh\" (UniqueName: \"kubernetes.io/projected/1f598db0-3077-44f7-ba00-40ab57c0a64b-kube-api-access-gw4nh\") pod \"maas-api-revoke-sub-26c136dc-l88v7\" (UID: \"1f598db0-3077-44f7-ba00-40ab57c0a64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c136dc-l88v7"
Nov 03 00:34:34.725189 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.725150    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c136dc-l88v7"
Nov 03 00:34:34.842679 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.842649    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:34:34.844186 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:34.844149    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:34:35.029642 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.029608    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26c136dc-l88v7"]
Nov 03 00:34:35.030250 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:34:35.030223    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f598db0_3077_44f7_ba00_40ab57c0a64b.slice/crio-d9a40ea032885ba37f4406d3bdf214166bbdcd533d2f6355206fd0973704e346 WatchSource:0}: Error finding container d9a40ea032885ba37f4406d3bdf214166bbdcd533d2f6355206fd0973704e346: Status 404 returned error can't find the container with id d9a40ea032885ba37f4406d3bdf214166bbdcd533d2f6355206fd0973704e346
Nov 03 00:34:35.571846 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.571679    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c136dc-l88v7" event={"ID":"1f598db0-3077-44f7-ba00-40ab57c0a64b","Type":"ContainerStarted","Data":"40b53c616e3fb04c117942fdab8af7b39de3e18a9618fdd07f670506636f7834"}
Nov 03 00:34:35.571846 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.571764    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c136dc-l88v7" event={"ID":"1f598db0-3077-44f7-ba00-40ab57c0a64b","Type":"ContainerStarted","Data":"d9a40ea032885ba37f4406d3bdf214166bbdcd533d2f6355206fd0973704e346"}
Nov 03 00:34:35.610227 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.610169    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c136dc-l88v7" podStartSLOduration=1.6101550470000001 podStartE2EDuration="1.610155047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:35.607149357 +0000 UTC m=+1758.486733400" watchObservedRunningTime="2026-11-03 00:34:35.610155047 +0000 UTC m=+1758.489739091"
Nov 03 00:34:35.784108 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.784074    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"]
Nov 03 00:34:35.784803 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.784782    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4f51e9c-5f2c-41df-a26f-4978e47bf980" containerName="istio-proxy"
Nov 03 00:34:35.788414 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.788396    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:35.808189 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.808158    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"]
Nov 03 00:34:35.816847 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.816819    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"]
Nov 03 00:34:35.817008 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.816993    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:35.828267 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.827315    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"]
Nov 03 00:34:35.961022 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.960975    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:35.961022 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.961017    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:35.961464 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.961049    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:35.961464 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.961171    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:35.961464 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.961205    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:35.961464 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.961257    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:35.961464 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.961333    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:35.961464 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.961402    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:35.961464 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.961429    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:35.961464 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.961462    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311cdd72-1e01-4590-ba0d-de18130c1d11-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:35.961748 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.961527    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:35.961748 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.961544    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:35.961748 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.961575    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:35.961748 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.961595    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25w9b\" (UniqueName: \"kubernetes.io/projected/311cdd72-1e01-4590-ba0d-de18130c1d11-kube-api-access-25w9b\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:35.961748 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.961711    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:35.961748 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.961731    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t79j2\" (UniqueName: \"kubernetes.io/projected/33ee11ba-035a-4f73-b194-12b3fdc9c32d-kube-api-access-t79j2\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:35.961991 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.961782    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:35.961991 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:35.961835    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:36.063251 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.063207    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:36.063251 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.063256    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25w9b\" (UniqueName: \"kubernetes.io/projected/311cdd72-1e01-4590-ba0d-de18130c1d11-kube-api-access-25w9b\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:36.063524 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.063299    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:36.063524 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.063318    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t79j2\" (UniqueName: \"kubernetes.io/projected/33ee11ba-035a-4f73-b194-12b3fdc9c32d-kube-api-access-t79j2\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:36.063524 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.063349    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:36.063524 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.063390    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:36.063524 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.063434    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:36.063524 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.063459    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:36.063524 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.063490    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:36.063956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.063544    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:36.063956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.063596    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:36.063956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.063638    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:36.063956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.063721    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:36.063956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.063777    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:36.063956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.063801    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:36.063956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.063810    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:36.063956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.063876    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311cdd72-1e01-4590-ba0d-de18130c1d11-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:36.063956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.063882    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:36.063956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.063930    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:36.063956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.063955    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:36.064548 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.064008    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:36.064548 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.064114    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:36.064548 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.064300    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:36.064548 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.064310    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:36.064548 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.064352    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:36.064873 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.064618    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:36.064873 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.064759    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311cdd72-1e01-4590-ba0d-de18130c1d11-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:36.064873 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.064813    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:36.066659 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.066631    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:36.066771 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.066680    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:36.066771 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.066698    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:36.066946 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.066922    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:36.082063 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.082009    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t79j2\" (UniqueName: \"kubernetes.io/projected/33ee11ba-035a-4f73-b194-12b3fdc9c32d-kube-api-access-t79j2\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:36.082650 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.082627    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25w9b\" (UniqueName: \"kubernetes.io/projected/311cdd72-1e01-4590-ba0d-de18130c1d11-kube-api-access-25w9b\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:36.082752 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.082733    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:36.082796 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.082777    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:36.120479 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.120428    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:36.154065 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.154036    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:36.454288 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.454262    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"]
Nov 03 00:34:36.455318 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:34:36.455290    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311cdd72_1e01_4590_ba0d_de18130c1d11.slice/crio-207b0d0e2fd2d96a80d81e863cca0289266bfc8c52af15b07db5a2c5efea3bbc WatchSource:0}: Error finding container 207b0d0e2fd2d96a80d81e863cca0289266bfc8c52af15b07db5a2c5efea3bbc: Status 404 returned error can't find the container with id 207b0d0e2fd2d96a80d81e863cca0289266bfc8c52af15b07db5a2c5efea3bbc
Nov 03 00:34:36.457962 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.457890    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:34:36.458076 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.457982    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:34:36.458076 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.458033    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:34:36.468616 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.468588    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"]
Nov 03 00:34:36.469580 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:34:36.469556    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ee11ba_035a_4f73_b194_12b3fdc9c32d.slice/crio-93f4bd87e633f2ba2f40b58cbe53c9f866c7e8ebf170247768bb448ac97310d2 WatchSource:0}: Error finding container 93f4bd87e633f2ba2f40b58cbe53c9f866c7e8ebf170247768bb448ac97310d2: Status 404 returned error can't find the container with id 93f4bd87e633f2ba2f40b58cbe53c9f866c7e8ebf170247768bb448ac97310d2
Nov 03 00:34:36.471824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.471793    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:34:36.471949 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.471860    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:34:36.471949 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.471925    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:34:36.583817 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.583790    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj" event={"ID":"311cdd72-1e01-4590-ba0d-de18130c1d11","Type":"ContainerStarted","Data":"50a465e4bf5596de7c7e7d6695e06b1668a821b6f59792da0b83803f39799346"}
Nov 03 00:34:36.583976 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.583846    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj" event={"ID":"311cdd72-1e01-4590-ba0d-de18130c1d11","Type":"ContainerStarted","Data":"207b0d0e2fd2d96a80d81e863cca0289266bfc8c52af15b07db5a2c5efea3bbc"}
Nov 03 00:34:36.586421 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.586392    5012 generic.go:356] "Generic (PLEG): container finished" podID="1f598db0-3077-44f7-ba00-40ab57c0a64b" containerID="40b53c616e3fb04c117942fdab8af7b39de3e18a9618fdd07f670506636f7834" exitCode=0
Nov 03 00:34:36.586551 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.586481    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c136dc-l88v7" event={"ID":"1f598db0-3077-44f7-ba00-40ab57c0a64b","Type":"ContainerDied","Data":"40b53c616e3fb04c117942fdab8af7b39de3e18a9618fdd07f670506636f7834"}
Nov 03 00:34:36.588612 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.588574    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7" event={"ID":"33ee11ba-035a-4f73-b194-12b3fdc9c32d","Type":"ContainerStarted","Data":"93f4bd87e633f2ba2f40b58cbe53c9f866c7e8ebf170247768bb448ac97310d2"}
Nov 03 00:34:36.626090 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.626034    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj" podStartSLOduration=1.626017357 podStartE2EDuration="1.626017357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:36.621360953 +0000 UTC m=+1759.500944998" watchObservedRunningTime="2026-11-03 00:34:36.626017357 +0000 UTC m=+1759.505601458"
Nov 03 00:34:36.672802 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:36.672515    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7" podStartSLOduration=1.6724963160000001 podStartE2EDuration="1.672496316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:36.668512872 +0000 UTC m=+1759.548096911" watchObservedRunningTime="2026-11-03 00:34:36.672496316 +0000 UTC m=+1759.552080360"
Nov 03 00:34:37.121165 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:37.121126    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:37.122683 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:37.122653    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 03 00:34:37.122836 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:37.122702    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj" podUID="311cdd72-1e01-4590-ba0d-de18130c1d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 03 00:34:37.154304 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:37.154272    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:37.155799 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:37.155772    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 03 00:34:37.155925 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:37.155829    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7" podUID="33ee11ba-035a-4f73-b194-12b3fdc9c32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 03 00:34:37.367744 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:34:37.367704    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:34:37.654947 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:37.654866    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7" event={"ID":"33ee11ba-035a-4f73-b194-12b3fdc9c32d","Type":"ContainerStarted","Data":"bcd2f45420a7b7ed278d3362edcbca6ed36d5d837848b7f3db4f7c554ba5b626"}
Nov 03 00:34:37.971689 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:37.971659    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c136dc-l88v7"
Nov 03 00:34:38.051151 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:38.051115    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f598db0-3077-44f7-ba00-40ab57c0a64b-kube-api-access-gw4nh\" (UniqueName: \"kubernetes.io/projected/1f598db0-3077-44f7-ba00-40ab57c0a64b-kube-api-access-gw4nh\") pod \"1f598db0-3077-44f7-ba00-40ab57c0a64b\" (UID: \"1f598db0-3077-44f7-ba00-40ab57c0a64b\") "
Nov 03 00:34:38.051360 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:38.051165    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f598db0-3077-44f7-ba00-40ab57c0a64b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f598db0-3077-44f7-ba00-40ab57c0a64b-maas-api-openshift-ca\") pod \"1f598db0-3077-44f7-ba00-40ab57c0a64b\" (UID: \"1f598db0-3077-44f7-ba00-40ab57c0a64b\") "
Nov 03 00:34:38.051360 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:38.051334    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f598db0-3077-44f7-ba00-40ab57c0a64b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f598db0-3077-44f7-ba00-40ab57c0a64b-maas-api-xks-ca\") pod \"1f598db0-3077-44f7-ba00-40ab57c0a64b\" (UID: \"1f598db0-3077-44f7-ba00-40ab57c0a64b\") "
Nov 03 00:34:38.052189 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:38.052164    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f598db0-3077-44f7-ba00-40ab57c0a64b-maas-api-openshift-ca" pod "1f598db0-3077-44f7-ba00-40ab57c0a64b" (UID: "1f598db0-3077-44f7-ba00-40ab57c0a64b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:38.054195 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:38.054165    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f598db0-3077-44f7-ba00-40ab57c0a64b-maas-api-xks-ca" pod "1f598db0-3077-44f7-ba00-40ab57c0a64b" (UID: "1f598db0-3077-44f7-ba00-40ab57c0a64b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:34:38.054195 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:38.054170    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f598db0-3077-44f7-ba00-40ab57c0a64b-kube-api-access-gw4nh" pod "1f598db0-3077-44f7-ba00-40ab57c0a64b" (UID: "1f598db0-3077-44f7-ba00-40ab57c0a64b"). InnerVolumeSpecName "kube-api-access-gw4nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:38.131933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:38.131211    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 03 00:34:38.131933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:38.131289    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj" podUID="311cdd72-1e01-4590-ba0d-de18130c1d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 03 00:34:38.153113 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:38.153080    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw4nh\" (UniqueName: \"kubernetes.io/projected/1f598db0-3077-44f7-ba00-40ab57c0a64b-kube-api-access-gw4nh\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:38.153113 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:38.153107    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f598db0-3077-44f7-ba00-40ab57c0a64b-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:38.153113 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:38.153119    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f598db0-3077-44f7-ba00-40ab57c0a64b-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:38.154641 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:38.154620    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 03 00:34:38.154705 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:38.154668    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7" podUID="33ee11ba-035a-4f73-b194-12b3fdc9c32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 03 00:34:38.664434 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:38.664398    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c136dc-l88v7" event={"ID":"1f598db0-3077-44f7-ba00-40ab57c0a64b","Type":"ContainerDied","Data":"d9a40ea032885ba37f4406d3bdf214166bbdcd533d2f6355206fd0973704e346"}
Nov 03 00:34:38.664630 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:38.664468    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9a40ea032885ba37f4406d3bdf214166bbdcd533d2f6355206fd0973704e346"
Nov 03 00:34:38.664630 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:38.664488    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c136dc-l88v7"
Nov 03 00:34:39.125935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:39.122467    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 03 00:34:39.125935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:39.122592    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj" podUID="311cdd72-1e01-4590-ba0d-de18130c1d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 03 00:34:39.161932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:39.161370    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 03 00:34:39.161932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:39.161433    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7" podUID="33ee11ba-035a-4f73-b194-12b3fdc9c32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 03 00:34:39.770052 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:39.769429    5012 generic.go:356] "Generic (PLEG): container finished" podID="14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1" containerID="d0977a786ea130ec6b0500906d0317de15430c37be67590cf99608ab11d7de1f" exitCode=0
Nov 03 00:34:39.770052 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:39.769591    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5" event={"ID":"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1","Type":"ContainerDied","Data":"d0977a786ea130ec6b0500906d0317de15430c37be67590cf99608ab11d7de1f"}
Nov 03 00:34:39.770052 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:39.769669    5012 scope.go:122] "RemoveContainer" containerID="0ddf9e2e137defc5edcc56e8d54690b8a3da2d0491340c0c8d72a41fba6c4332"
Nov 03 00:34:40.367934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:40.199387    5012 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:40.367934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:40.329406    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 03 00:34:40.367934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:40.360638    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj" podUID="311cdd72-1e01-4590-ba0d-de18130c1d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 03 00:34:40.367934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:40.330119    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 03 00:34:40.367934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:40.360705    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7" podUID="33ee11ba-035a-4f73-b194-12b3fdc9c32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 03 00:34:40.367934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:40.360781    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Nov 03 00:34:40.384819 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:40.384771    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:40.482528 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:40.482480    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 03 00:34:40.676707 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:40.676612    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:34:40.869937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:40.857657    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:40.996837 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:40.996751    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-podinfo\") pod \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") "
Nov 03 00:34:41.082937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.082569    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-kube-api-access-69stt\" (UniqueName: \"kubernetes.io/projected/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-kube-api-access-69stt\") pod \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") "
Nov 03 00:34:41.082937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.082705    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-credential-socket\") pod \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") "
Nov 03 00:34:41.082937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.082743    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-envoy\") pod \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") "
Nov 03 00:34:41.082937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.082811    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-token\" (UniqueName: \"kubernetes.io/projected/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-token\") pod \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") "
Nov 03 00:34:41.082937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.082857    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istiod-ca-cert\") pod \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") "
Nov 03 00:34:41.083335 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.082990    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-workload-certs\") pod \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") "
Nov 03 00:34:41.083335 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.083034    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-workload-socket\") pod \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") "
Nov 03 00:34:41.083335 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.083059    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-data\") pod \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\" (UID: \"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1\") "
Nov 03 00:34:41.101940 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.035331    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-podinfo" pod "14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1" (UID: "14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:34:41.101940 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.027354    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5" event={"ID":"14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1","Type":"ContainerDied","Data":"03bd0adf1baf4dcd9d784a44c648a3ae94d99367ccc1ef768b6bc5d153df4962"}
Nov 03 00:34:41.101940 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.101057    5012 scope.go:122] "RemoveContainer" containerID="d0977a786ea130ec6b0500906d0317de15430c37be67590cf99608ab11d7de1f"
Nov 03 00:34:41.105938 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.027437    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"
Nov 03 00:34:41.106300 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.106271    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istiod-ca-cert" pod "14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1" (UID: "14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:41.111626 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.111588    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-workload-certs" pod "14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1" (UID: "14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:34:41.115050 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.115015    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-workload-socket" pod "14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1" (UID: "14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:34:41.121121 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.121077    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-data" pod "14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1" (UID: "14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:34:41.121766 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.121744    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-credential-socket" pod "14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1" (UID: "14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:34:41.125861 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.125830    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 03 00:34:41.126080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.126037    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj" podUID="311cdd72-1e01-4590-ba0d-de18130c1d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 03 00:34:41.176498 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.176449    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 03 00:34:41.176778 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.176740    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7" podUID="33ee11ba-035a-4f73-b194-12b3fdc9c32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 03 00:34:41.177950 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.177929    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:41.178102 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.178086    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:41.178249 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.178232    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:41.178384 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.178368    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:41.178516 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.178498    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:41.183384 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.183361    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:41.183606 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.183564    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:41.374387 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.369256    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Nov 03 00:34:41.413942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.411674    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:34:41.413942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.412900    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:41.413942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.413412    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:34:41.506555 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.506501    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 03 00:34:41.507546 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.507523    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:34:41.587139 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.587037    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-envoy" pod "14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1" (UID: "14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:34:41.619963 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.610079    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-kube-api-access-69stt" pod "14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1" (UID: "14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1"). InnerVolumeSpecName "kube-api-access-69stt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:41.717936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.679179    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:41.764351 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.764320    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69stt\" (UniqueName: \"kubernetes.io/projected/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-kube-api-access-69stt\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:41.790368 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.749121    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-token" pod "14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1" (UID: "14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:41.875934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.868836    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:41.876278 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.876251    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:34:41.927935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:41.895136    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:42.247080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:42.193142    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 03 00:34:42.247080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:42.193724    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:34:42.384934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:42.379312    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:42.711935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:42.610707    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:43.168700 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:42.743539    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:43.169086 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:43.047058    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:43.169268 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:43.147824    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 03 00:34:43.169406 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:43.169377    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj" podUID="311cdd72-1e01-4590-ba0d-de18130c1d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 03 00:34:43.421988 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:43.415968    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:43.519038 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:43.518983    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:34:43.519301 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:43.519269    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9" podUID="81f130bb-d23a-4a79-b458-86a198845567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:43.532739 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:43.532703    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 03 00:34:43.550853 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:43.550811    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7" podUID="33ee11ba-035a-4f73-b194-12b3fdc9c32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 03 00:34:43.579508 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:43.551675    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:34:43.579508 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:43.574536    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm" podUID="a5e7d5fd-29e5-4719-8239-922e7a3aeeed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:43.579508 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:43.557077    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 03 00:34:43.579508 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:43.574593    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj" podUID="311cdd72-1e01-4590-ba0d-de18130c1d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 03 00:34:43.579508 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:43.576620    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:43.580109 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:43.580081    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 03 00:34:43.580264 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:43.580240    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7" podUID="33ee11ba-035a-4f73-b194-12b3fdc9c32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 03 00:34:44.073960 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:44.058981    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"]
Nov 03 00:34:44.135932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:44.110645    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-94gf5"]
Nov 03 00:34:44.161889 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:44.139043    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 03 00:34:44.162141 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:44.160488    5012 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": context deadline exceeded"
Nov 03 00:34:44.182558 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:44.180283    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:44.182558 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:44.181329    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:44.229930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:44.160443    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 03 00:34:44.229930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:44.210015    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7" podUID="33ee11ba-035a-4f73-b194-12b3fdc9c32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 03 00:34:44.426293 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:44.385562    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj" podUID="311cdd72-1e01-4590-ba0d-de18130c1d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 03 00:34:44.426293 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:44.422998    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:44.542934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:44.542143    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded"
Nov 03 00:34:44.592942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:44.592005    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:44.734987 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:34:44.727376    5012 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f45c1d_a84e_4e9b_bec1_6df5d0d1eab1.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f45c1d_a84e_4e9b_bec1_6df5d0d1eab1.slice/crio-03bd0adf1baf4dcd9d784a44c648a3ae94d99367ccc1ef768b6bc5d153df4962\": RecentStats: unable to find data in memory cache]"
Nov 03 00:34:44.784938 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:44.776683    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:34:44.785725 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:44.785698    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:34:45.057564 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:44.979751    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:34:45.084256 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:45.084227    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:34:45.147069 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:45.147035    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 03 00:34:45.148925 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:45.148877    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj" podUID="311cdd72-1e01-4590-ba0d-de18130c1d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 03 00:34:45.150811 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:45.150759    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:34:45.155297 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:45.155259    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 03 00:34:45.155471 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:45.155449    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7" podUID="33ee11ba-035a-4f73-b194-12b3fdc9c32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 03 00:34:45.256121 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:45.256057    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1" path="/var/lib/kubelet/pods/14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1/volumes"
Nov 03 00:34:46.122347 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:46.122307    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 03 00:34:46.123007 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:46.122973    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj" podUID="311cdd72-1e01-4590-ba0d-de18130c1d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 03 00:34:46.154979 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:46.154949    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 03 00:34:46.155118 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:46.154997    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7" podUID="33ee11ba-035a-4f73-b194-12b3fdc9c32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 03 00:34:47.121056 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:47.121017    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 03 00:34:47.121234 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:47.121082    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj" podUID="311cdd72-1e01-4590-ba0d-de18130c1d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 03 00:34:47.154513 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:47.154477    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 03 00:34:47.154889 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:47.154540    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7" podUID="33ee11ba-035a-4f73-b194-12b3fdc9c32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 03 00:34:49.175057 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:49.175023    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:34:49.175546 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:49.175075    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj" podUID="311cdd72-1e01-4590-ba0d-de18130c1d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:49.196413 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:49.196374    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:34:49.196641 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:49.196615    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7" podUID="33ee11ba-035a-4f73-b194-12b3fdc9c32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:49.554109 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:49.554021    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:49.555162 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:49.555138    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:49.558042 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:49.558018    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:34:49.642788 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:49.642759    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:49.643523 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:49.643476    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:49.644326 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:49.644296    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:34:50.632444 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:50.632412    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9bb9669-q6jbv"]
Nov 03 00:34:50.633033 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:50.633020    5012 cpu_manager.go:414] "containerMap: removing container" podUID="1f598db0-3077-44f7-ba00-40ab57c0a64b" containerName="revoke-keys"
Nov 03 00:34:50.633092 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:50.633034    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f598db0-3077-44f7-ba00-40ab57c0a64b" containerName="revoke-keys"
Nov 03 00:34:50.633092 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:50.633051    5012 cpu_manager.go:414] "containerMap: removing container" podUID="14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1" containerName="istio-proxy"
Nov 03 00:34:50.633092 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:50.633056    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1" containerName="istio-proxy"
Nov 03 00:34:50.633092 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:50.633067    5012 cpu_manager.go:414] "containerMap: removing container" podUID="14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1" containerName="istio-proxy"
Nov 03 00:34:50.633092 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:50.633072    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1" containerName="istio-proxy"
Nov 03 00:34:50.633268 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:50.633152    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f598db0-3077-44f7-ba00-40ab57c0a64b" containerName="revoke-keys"
Nov 03 00:34:50.633268 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:50.633163    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1" containerName="istio-proxy"
Nov 03 00:34:50.633268 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:50.633170    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="14f45c1d-a84e-4e9b-bec1-6df5d0d1eab1" containerName="istio-proxy"
Nov 03 00:34:50.638347 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:50.638327    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bb9669-q6jbv"
Nov 03 00:34:50.654502 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:50.654480    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9bb9669-q6jbv"]
Nov 03 00:34:50.678845 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:50.665303    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:34:50.678845 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:50.665395    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:34:50.692283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:50.692256    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41e7eae9-ad3b-4011-9f82-22e4436ceb79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9bb9669-q6jbv\" (UID: \"41e7eae9-ad3b-4011-9f82-22e4436ceb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bb9669-q6jbv"
Nov 03 00:34:50.692431 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:50.692287    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47dtp\" (UniqueName: \"kubernetes.io/projected/41e7eae9-ad3b-4011-9f82-22e4436ceb79-kube-api-access-47dtp\") pod \"maas-api-revoke-sub-e9bb9669-q6jbv\" (UID: \"41e7eae9-ad3b-4011-9f82-22e4436ceb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bb9669-q6jbv"
Nov 03 00:34:50.692534 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:50.692511    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41e7eae9-ad3b-4011-9f82-22e4436ceb79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9bb9669-q6jbv\" (UID: \"41e7eae9-ad3b-4011-9f82-22e4436ceb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bb9669-q6jbv"
Nov 03 00:34:50.793621 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:50.793594    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41e7eae9-ad3b-4011-9f82-22e4436ceb79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9bb9669-q6jbv\" (UID: \"41e7eae9-ad3b-4011-9f82-22e4436ceb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bb9669-q6jbv"
Nov 03 00:34:50.793621 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:50.793622    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47dtp\" (UniqueName: \"kubernetes.io/projected/41e7eae9-ad3b-4011-9f82-22e4436ceb79-kube-api-access-47dtp\") pod \"maas-api-revoke-sub-e9bb9669-q6jbv\" (UID: \"41e7eae9-ad3b-4011-9f82-22e4436ceb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bb9669-q6jbv"
Nov 03 00:34:50.793799 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:50.793692    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41e7eae9-ad3b-4011-9f82-22e4436ceb79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9bb9669-q6jbv\" (UID: \"41e7eae9-ad3b-4011-9f82-22e4436ceb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bb9669-q6jbv"
Nov 03 00:34:50.794253 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:50.794230    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41e7eae9-ad3b-4011-9f82-22e4436ceb79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9bb9669-q6jbv\" (UID: \"41e7eae9-ad3b-4011-9f82-22e4436ceb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bb9669-q6jbv"
Nov 03 00:34:50.796089 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:50.796073    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41e7eae9-ad3b-4011-9f82-22e4436ceb79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9bb9669-q6jbv\" (UID: \"41e7eae9-ad3b-4011-9f82-22e4436ceb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bb9669-q6jbv"
Nov 03 00:34:50.811029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:50.811005    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47dtp\" (UniqueName: \"kubernetes.io/projected/41e7eae9-ad3b-4011-9f82-22e4436ceb79-kube-api-access-47dtp\") pod \"maas-api-revoke-sub-e9bb9669-q6jbv\" (UID: \"41e7eae9-ad3b-4011-9f82-22e4436ceb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bb9669-q6jbv"
Nov 03 00:34:50.955765 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:50.955709    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bb9669-q6jbv"
Nov 03 00:34:51.275899 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:34:51.275870    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e7eae9_ad3b_4011_9f82_22e4436ceb79.slice/crio-34768c2e873182a321fadcca77bd7838b4790d052670453879534b8eddd3d8e8 WatchSource:0}: Error finding container 34768c2e873182a321fadcca77bd7838b4790d052670453879534b8eddd3d8e8: Status 404 returned error can't find the container with id 34768c2e873182a321fadcca77bd7838b4790d052670453879534b8eddd3d8e8
Nov 03 00:34:51.276019 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:51.276003    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9bb9669-q6jbv"]
Nov 03 00:34:51.825172 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:51.825129    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bb9669-q6jbv" event={"ID":"41e7eae9-ad3b-4011-9f82-22e4436ceb79","Type":"ContainerStarted","Data":"ac62147ea9623a20f7827cde25fe97c2d5ff50c336f66299047aa7d3ed46c43d"}
Nov 03 00:34:51.825594 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:51.825201    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bb9669-q6jbv" event={"ID":"41e7eae9-ad3b-4011-9f82-22e4436ceb79","Type":"ContainerStarted","Data":"34768c2e873182a321fadcca77bd7838b4790d052670453879534b8eddd3d8e8"}
Nov 03 00:34:51.863571 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:51.863521    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bb9669-q6jbv" podStartSLOduration=1.8635085679999999 podStartE2EDuration="1.863508568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:51.86058559 +0000 UTC m=+1774.740169670" watchObservedRunningTime="2026-11-03 00:34:51.863508568 +0000 UTC m=+1774.743092611"
Nov 03 00:34:52.835218 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:52.835186    5012 generic.go:356] "Generic (PLEG): container finished" podID="41e7eae9-ad3b-4011-9f82-22e4436ceb79" containerID="ac62147ea9623a20f7827cde25fe97c2d5ff50c336f66299047aa7d3ed46c43d" exitCode=0
Nov 03 00:34:52.835632 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:52.835273    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bb9669-q6jbv" event={"ID":"41e7eae9-ad3b-4011-9f82-22e4436ceb79","Type":"ContainerDied","Data":"ac62147ea9623a20f7827cde25fe97c2d5ff50c336f66299047aa7d3ed46c43d"}
Nov 03 00:34:54.086408 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:54.086374    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bb9669-q6jbv"
Nov 03 00:34:54.184836 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:54.184758    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41e7eae9-ad3b-4011-9f82-22e4436ceb79-kube-api-access-47dtp\" (UniqueName: \"kubernetes.io/projected/41e7eae9-ad3b-4011-9f82-22e4436ceb79-kube-api-access-47dtp\") pod \"41e7eae9-ad3b-4011-9f82-22e4436ceb79\" (UID: \"41e7eae9-ad3b-4011-9f82-22e4436ceb79\") "
Nov 03 00:34:54.185000 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:54.184935    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41e7eae9-ad3b-4011-9f82-22e4436ceb79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41e7eae9-ad3b-4011-9f82-22e4436ceb79-maas-api-openshift-ca\") pod \"41e7eae9-ad3b-4011-9f82-22e4436ceb79\" (UID: \"41e7eae9-ad3b-4011-9f82-22e4436ceb79\") "
Nov 03 00:34:54.185071 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:54.184996    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41e7eae9-ad3b-4011-9f82-22e4436ceb79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41e7eae9-ad3b-4011-9f82-22e4436ceb79-maas-api-xks-ca\") pod \"41e7eae9-ad3b-4011-9f82-22e4436ceb79\" (UID: \"41e7eae9-ad3b-4011-9f82-22e4436ceb79\") "
Nov 03 00:34:54.185325 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:54.185298    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e7eae9-ad3b-4011-9f82-22e4436ceb79-maas-api-openshift-ca" pod "41e7eae9-ad3b-4011-9f82-22e4436ceb79" (UID: "41e7eae9-ad3b-4011-9f82-22e4436ceb79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:54.185476 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:54.185458    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41e7eae9-ad3b-4011-9f82-22e4436ceb79-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:54.186960 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:54.186899    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e7eae9-ad3b-4011-9f82-22e4436ceb79-kube-api-access-47dtp" pod "41e7eae9-ad3b-4011-9f82-22e4436ceb79" (UID: "41e7eae9-ad3b-4011-9f82-22e4436ceb79"). InnerVolumeSpecName "kube-api-access-47dtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:54.187208 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:54.187190    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41e7eae9-ad3b-4011-9f82-22e4436ceb79-maas-api-xks-ca" pod "41e7eae9-ad3b-4011-9f82-22e4436ceb79" (UID: "41e7eae9-ad3b-4011-9f82-22e4436ceb79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:34:54.286460 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:54.286427    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41e7eae9-ad3b-4011-9f82-22e4436ceb79-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:54.286460 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:54.286458    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47dtp\" (UniqueName: \"kubernetes.io/projected/41e7eae9-ad3b-4011-9f82-22e4436ceb79-kube-api-access-47dtp\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:54.901001 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:54.900973    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bb9669-q6jbv"
Nov 03 00:34:54.901181 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:54.901020    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bb9669-q6jbv" event={"ID":"41e7eae9-ad3b-4011-9f82-22e4436ceb79","Type":"ContainerDied","Data":"34768c2e873182a321fadcca77bd7838b4790d052670453879534b8eddd3d8e8"}
Nov 03 00:34:54.901181 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:54.901070    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34768c2e873182a321fadcca77bd7838b4790d052670453879534b8eddd3d8e8"
Nov 03 00:34:56.788184 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:56.788147    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw"]
Nov 03 00:34:56.788778 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:56.788374    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw" podUID="0c980a86-f763-4b5a-90ce-6961c1d4e947" containerName="maas-api" containerID="cri-o://8d99d90a95c769a6624e5cb96206512c2077f85237175f6a027ad0c1d4a80a33" gracePeriod=30
Nov 03 00:34:56.917239 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:56.917211    5012 generic.go:356] "Generic (PLEG): container finished" podID="0c980a86-f763-4b5a-90ce-6961c1d4e947" containerID="8d99d90a95c769a6624e5cb96206512c2077f85237175f6a027ad0c1d4a80a33" exitCode=0
Nov 03 00:34:56.917349 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:56.917279    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw" event={"ID":"0c980a86-f763-4b5a-90ce-6961c1d4e947","Type":"ContainerDied","Data":"8d99d90a95c769a6624e5cb96206512c2077f85237175f6a027ad0c1d4a80a33"}
Nov 03 00:34:57.232347 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:57.232313    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw"
Nov 03 00:34:57.325661 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:57.325631    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c980a86-f763-4b5a-90ce-6961c1d4e947-kube-api-access-75gw5\" (UniqueName: \"kubernetes.io/projected/0c980a86-f763-4b5a-90ce-6961c1d4e947-kube-api-access-75gw5\") pod \"0c980a86-f763-4b5a-90ce-6961c1d4e947\" (UID: \"0c980a86-f763-4b5a-90ce-6961c1d4e947\") "
Nov 03 00:34:57.325809 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:57.325764    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c980a86-f763-4b5a-90ce-6961c1d4e947-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c980a86-f763-4b5a-90ce-6961c1d4e947-metrics-certs\") pod \"0c980a86-f763-4b5a-90ce-6961c1d4e947\" (UID: \"0c980a86-f763-4b5a-90ce-6961c1d4e947\") "
Nov 03 00:34:57.325871 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:57.325817    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c980a86-f763-4b5a-90ce-6961c1d4e947-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c980a86-f763-4b5a-90ce-6961c1d4e947-maas-api-tls\") pod \"0c980a86-f763-4b5a-90ce-6961c1d4e947\" (UID: \"0c980a86-f763-4b5a-90ce-6961c1d4e947\") "
Nov 03 00:34:57.326018 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:57.325998    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c980a86-f763-4b5a-90ce-6961c1d4e947-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c980a86-f763-4b5a-90ce-6961c1d4e947-openshift-service-ca\") pod \"0c980a86-f763-4b5a-90ce-6961c1d4e947\" (UID: \"0c980a86-f763-4b5a-90ce-6961c1d4e947\") "
Nov 03 00:34:57.326460 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:57.326424    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c980a86-f763-4b5a-90ce-6961c1d4e947-openshift-service-ca" pod "0c980a86-f763-4b5a-90ce-6961c1d4e947" (UID: "0c980a86-f763-4b5a-90ce-6961c1d4e947"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:57.326783 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:57.326751    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c980a86-f763-4b5a-90ce-6961c1d4e947-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:57.328843 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:57.328814    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c980a86-f763-4b5a-90ce-6961c1d4e947-kube-api-access-75gw5" pod "0c980a86-f763-4b5a-90ce-6961c1d4e947" (UID: "0c980a86-f763-4b5a-90ce-6961c1d4e947"). InnerVolumeSpecName "kube-api-access-75gw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:57.337975 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:57.333262    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c980a86-f763-4b5a-90ce-6961c1d4e947-metrics-certs" pod "0c980a86-f763-4b5a-90ce-6961c1d4e947" (UID: "0c980a86-f763-4b5a-90ce-6961c1d4e947"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:34:57.337975 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:57.333337    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c980a86-f763-4b5a-90ce-6961c1d4e947-maas-api-tls" pod "0c980a86-f763-4b5a-90ce-6961c1d4e947" (UID: "0c980a86-f763-4b5a-90ce-6961c1d4e947"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:34:57.428507 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:57.428418    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75gw5\" (UniqueName: \"kubernetes.io/projected/0c980a86-f763-4b5a-90ce-6961c1d4e947-kube-api-access-75gw5\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:57.428507 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:57.428454    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c980a86-f763-4b5a-90ce-6961c1d4e947-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:57.428507 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:57.428466    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c980a86-f763-4b5a-90ce-6961c1d4e947-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:34:57.926678 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:57.926649    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw"
Nov 03 00:34:57.927203 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:57.926647    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw" event={"ID":"0c980a86-f763-4b5a-90ce-6961c1d4e947","Type":"ContainerDied","Data":"de75e38c89eb0442be6739fbdb2627d05e7666f6103116b6fc84ac3a1cd43118"}
Nov 03 00:34:57.927203 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:57.926794    5012 scope.go:122] "RemoveContainer" containerID="8d99d90a95c769a6624e5cb96206512c2077f85237175f6a027ad0c1d4a80a33"
Nov 03 00:34:57.976234 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:57.976209    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw"]
Nov 03 00:34:57.988117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:57.988096    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a19e67-77bdf9b89b-lvfrw"]
Nov 03 00:34:59.243955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:34:59.243897    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c980a86-f763-4b5a-90ce-6961c1d4e947" path="/var/lib/kubelet/pods/0c980a86-f763-4b5a-90ce-6961c1d4e947/volumes"
Nov 03 00:35:00.619455 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.619419    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"]
Nov 03 00:35:00.620210 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.620187    5012 cpu_manager.go:414] "containerMap: removing container" podUID="0c980a86-f763-4b5a-90ce-6961c1d4e947" containerName="maas-api"
Nov 03 00:35:00.620210 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.620206    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c980a86-f763-4b5a-90ce-6961c1d4e947" containerName="maas-api"
Nov 03 00:35:00.620444 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.620227    5012 cpu_manager.go:414] "containerMap: removing container" podUID="41e7eae9-ad3b-4011-9f82-22e4436ceb79" containerName="revoke-keys"
Nov 03 00:35:00.620444 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.620234    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41e7eae9-ad3b-4011-9f82-22e4436ceb79" containerName="revoke-keys"
Nov 03 00:35:00.620444 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.620345    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="41e7eae9-ad3b-4011-9f82-22e4436ceb79" containerName="revoke-keys"
Nov 03 00:35:00.620444 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.620365    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c980a86-f763-4b5a-90ce-6961c1d4e947" containerName="maas-api"
Nov 03 00:35:00.625629 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.625609    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.643169 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.643136    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-e62935-openshift-default-dockercfg-bs2xk\""
Nov 03 00:35:00.660061 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.660031    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"]
Nov 03 00:35:00.664289 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.664261    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-workload-certs\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.664457 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.664431    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5795e973-761d-43af-91ea-0451024c0d42-istio-token\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.664533 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.664474    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-credential-socket\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.664623 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.664604    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-istio-envoy\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.664684 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.664653    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-istio-data\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.664761 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.664739    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82mgl\" (UniqueName: \"kubernetes.io/projected/5795e973-761d-43af-91ea-0451024c0d42-kube-api-access-82mgl\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.664858 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.664830    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-workload-socket\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.664943 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.664887    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5795e973-761d-43af-91ea-0451024c0d42-istiod-ca-cert\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.664943 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.664937    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5795e973-761d-43af-91ea-0451024c0d42-istio-podinfo\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.766897 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.766859    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-istio-envoy\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.767097 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.766946    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-istio-data\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.767097 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.766983    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82mgl\" (UniqueName: \"kubernetes.io/projected/5795e973-761d-43af-91ea-0451024c0d42-kube-api-access-82mgl\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.767097 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.767037    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-workload-socket\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.767097 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.767093    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5795e973-761d-43af-91ea-0451024c0d42-istiod-ca-cert\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.767345 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.767127    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5795e973-761d-43af-91ea-0451024c0d42-istio-podinfo\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.767345 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.767224    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-workload-certs\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.767345 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.767328    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5795e973-761d-43af-91ea-0451024c0d42-istio-token\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.767522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.767359    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-credential-socket\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.767522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.767492    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-istio-data\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.767832 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.767805    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-credential-socket\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.768178 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.768155    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-workload-certs\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.768327 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.768183    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5795e973-761d-43af-91ea-0451024c0d42-istiod-ca-cert\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.768670 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.768646    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-workload-socket\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.770709 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.770687    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-istio-envoy\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.771334 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.771308    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5795e973-761d-43af-91ea-0451024c0d42-istio-podinfo\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.786157 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.786124    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5795e973-761d-43af-91ea-0451024c0d42-istio-token\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.786329 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.786304    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82mgl\" (UniqueName: \"kubernetes.io/projected/5795e973-761d-43af-91ea-0451024c0d42-kube-api-access-82mgl\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:00.954011 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:00.945776    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:01.176947 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:01.176904    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d648ba4-9s9rs"]
Nov 03 00:35:01.182410 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:01.182383    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d648ba4-9s9rs"
Nov 03 00:35:01.196794 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:01.196352    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:35:01.196794 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:01.196494    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 00:35:01.196794 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:01.196517    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 00:35:01.196794 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:01.196565    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:35:01.199943 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:01.199885    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d648ba4-9s9rs"]
Nov 03 00:35:01.289943 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:01.289832    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c48696-42c4-4710-beb9-434cfa1ed809-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d648ba4-9s9rs\" (UID: \"82c48696-42c4-4710-beb9-434cfa1ed809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d648ba4-9s9rs"
Nov 03 00:35:01.289943 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:01.289888    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c48696-42c4-4710-beb9-434cfa1ed809-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d648ba4-9s9rs\" (UID: \"82c48696-42c4-4710-beb9-434cfa1ed809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d648ba4-9s9rs"
Nov 03 00:35:01.290123 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:01.289946    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lmg7\" (UniqueName: \"kubernetes.io/projected/82c48696-42c4-4710-beb9-434cfa1ed809-kube-api-access-8lmg7\") pod \"maas-api-revoke-sub-8d648ba4-9s9rs\" (UID: \"82c48696-42c4-4710-beb9-434cfa1ed809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d648ba4-9s9rs"
Nov 03 00:35:01.377581 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:01.377553    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"]
Nov 03 00:35:01.378921 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:35:01.378883    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5795e973_761d_43af_91ea_0451024c0d42.slice/crio-b099b0c658cfd5a6e6491edf03d71bae61814ef61a8d9696334b9166b1d3d2fe WatchSource:0}: Error finding container b099b0c658cfd5a6e6491edf03d71bae61814ef61a8d9696334b9166b1d3d2fe: Status 404 returned error can't find the container with id b099b0c658cfd5a6e6491edf03d71bae61814ef61a8d9696334b9166b1d3d2fe
Nov 03 00:35:01.382128 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:01.382091    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:01.382215 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:01.382171    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:01.382286 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:01.382252    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:01.391418 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:01.391388    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c48696-42c4-4710-beb9-434cfa1ed809-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d648ba4-9s9rs\" (UID: \"82c48696-42c4-4710-beb9-434cfa1ed809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d648ba4-9s9rs"
Nov 03 00:35:01.391508 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:01.391438    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c48696-42c4-4710-beb9-434cfa1ed809-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d648ba4-9s9rs\" (UID: \"82c48696-42c4-4710-beb9-434cfa1ed809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d648ba4-9s9rs"
Nov 03 00:35:01.391578 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:01.391557    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lmg7\" (UniqueName: \"kubernetes.io/projected/82c48696-42c4-4710-beb9-434cfa1ed809-kube-api-access-8lmg7\") pod \"maas-api-revoke-sub-8d648ba4-9s9rs\" (UID: \"82c48696-42c4-4710-beb9-434cfa1ed809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d648ba4-9s9rs"
Nov 03 00:35:01.392149 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:01.392127    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c48696-42c4-4710-beb9-434cfa1ed809-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d648ba4-9s9rs\" (UID: \"82c48696-42c4-4710-beb9-434cfa1ed809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d648ba4-9s9rs"
Nov 03 00:35:01.393993 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:01.393973    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c48696-42c4-4710-beb9-434cfa1ed809-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d648ba4-9s9rs\" (UID: \"82c48696-42c4-4710-beb9-434cfa1ed809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d648ba4-9s9rs"
Nov 03 00:35:01.411287 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:01.411267    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lmg7\" (UniqueName: \"kubernetes.io/projected/82c48696-42c4-4710-beb9-434cfa1ed809-kube-api-access-8lmg7\") pod \"maas-api-revoke-sub-8d648ba4-9s9rs\" (UID: \"82c48696-42c4-4710-beb9-434cfa1ed809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d648ba4-9s9rs"
Nov 03 00:35:01.500325 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:01.500021    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d648ba4-9s9rs"
Nov 03 00:35:01.838323 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:01.838285    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d648ba4-9s9rs"]
Nov 03 00:35:01.840363 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:35:01.840326    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82c48696_42c4_4710_beb9_434cfa1ed809.slice/crio-c64a7db2245cbeaacb87ef8a0849cf3925c514a8d3b45162a0f90285e4b93a7e WatchSource:0}: Error finding container c64a7db2245cbeaacb87ef8a0849cf3925c514a8d3b45162a0f90285e4b93a7e: Status 404 returned error can't find the container with id c64a7db2245cbeaacb87ef8a0849cf3925c514a8d3b45162a0f90285e4b93a7e
Nov 03 00:35:02.040205 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:02.040160    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d648ba4-9s9rs" event={"ID":"82c48696-42c4-4710-beb9-434cfa1ed809","Type":"ContainerStarted","Data":"c64a7db2245cbeaacb87ef8a0849cf3925c514a8d3b45162a0f90285e4b93a7e"}
Nov 03 00:35:02.042136 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:02.042101    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94" event={"ID":"5795e973-761d-43af-91ea-0451024c0d42","Type":"ContainerStarted","Data":"793b72e6c7c6c379be340ef4e6637111ace89e4ca35f2443ed4fcaf265e01488"}
Nov 03 00:35:02.042314 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:02.042157    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94" event={"ID":"5795e973-761d-43af-91ea-0451024c0d42","Type":"ContainerStarted","Data":"b099b0c658cfd5a6e6491edf03d71bae61814ef61a8d9696334b9166b1d3d2fe"}
Nov 03 00:35:02.101128 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:02.098045    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94" podStartSLOduration=2.098021585 podStartE2EDuration="2.098021585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:02.088444533 +0000 UTC m=+1784.968028638" watchObservedRunningTime="2026-11-03 00:35:02.098021585 +0000 UTC m=+1784.977605631"
Nov 03 00:35:02.951935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:02.951347    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:03.116336 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:03.116303    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:03.140952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:03.122799    5012 generic.go:356] "Generic (PLEG): container finished" podID="82c48696-42c4-4710-beb9-434cfa1ed809" containerID="fe825e977a05c3ce2c668b568e3e156f97817eed629d58d9e73ad34a7ab39d86" exitCode=0
Nov 03 00:35:03.140952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:03.124682    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d648ba4-9s9rs" event={"ID":"82c48696-42c4-4710-beb9-434cfa1ed809","Type":"ContainerDied","Data":"fe825e977a05c3ce2c668b568e3e156f97817eed629d58d9e73ad34a7ab39d86"}
Nov 03 00:35:03.140952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:03.124775    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:03.140952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:03.130509    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:04.127663 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:04.127015    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:04.212940 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:04.172619    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:04.212940 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:04.197070    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:04.574205 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:04.574015    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"]
Nov 03 00:35:04.574983 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:04.574943    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln" podUID="6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5" containerName="istio-proxy" containerID="cri-o://8d1fb92ef148cf95773de678772db9c73a61acad6dff5adf202199fd22a23974" gracePeriod=30
Nov 03 00:35:04.575447 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:04.575415    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Nov 03 00:35:04.688934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:04.688467    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:04.920937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:04.896495    5012 patch_prober.go:29] interesting pod/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:35:04.920937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:04.896558    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln" podUID="6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:35:05.134065 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:05.134009    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:05.153702 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:05.153652    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:35:05.178254 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:05.177587    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:05.179036 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:05.178994    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:35:05.199591 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:05.198063    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded"
Nov 03 00:35:05.199591 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:05.198721    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:35:05.200466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:05.200438    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:35:05.200709 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:05.200670    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx" podUID="2431e1a4-b1df-4516-a2bf-680be875a7d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:05.630943 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:05.627294    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:35:05.630943 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:05.627370    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7" podUID="33ee11ba-035a-4f73-b194-12b3fdc9c32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:05.638068 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:05.638009    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:05.641938 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:05.638814    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:35:05.707224 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:05.707167    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:05.737938 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:05.708881    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:35:06.024935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:05.781059    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:35:06.060935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:05.966015    5012 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/health\": context deadline exceeded"
Nov 03 00:35:06.060935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:06.056967    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj" podUID="311cdd72-1e01-4590-ba0d-de18130c1d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:06.166952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:06.164221    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:06.192947 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:06.190046    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 03 00:35:06.218936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:06.218484    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:06.650962 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:06.639977    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz"]
Nov 03 00:35:06.675965 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:06.665989    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Nov 03 00:35:06.711130 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:06.711012    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:06.814936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:06.786171    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d648ba4-9s9rs"
Nov 03 00:35:06.814936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:06.813168    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz"]
Nov 03 00:35:06.814936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:06.813307    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz"
Nov 03 00:35:06.882404 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:06.882363    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 00:35:06.932313 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:06.890073    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-x7pjw\""
Nov 03 00:35:06.932557 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:06.906009    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-065de0-serving-cert\""
Nov 03 00:35:06.968040 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:06.957402    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82c48696-42c4-4710-beb9-434cfa1ed809-kube-api-access-8lmg7\" (UniqueName: \"kubernetes.io/projected/82c48696-42c4-4710-beb9-434cfa1ed809-kube-api-access-8lmg7\") pod \"82c48696-42c4-4710-beb9-434cfa1ed809\" (UID: \"82c48696-42c4-4710-beb9-434cfa1ed809\") "
Nov 03 00:35:06.968040 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:06.957558    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82c48696-42c4-4710-beb9-434cfa1ed809-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c48696-42c4-4710-beb9-434cfa1ed809-maas-api-openshift-ca\") pod \"82c48696-42c4-4710-beb9-434cfa1ed809\" (UID: \"82c48696-42c4-4710-beb9-434cfa1ed809\") "
Nov 03 00:35:06.968040 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:06.957598    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82c48696-42c4-4710-beb9-434cfa1ed809-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c48696-42c4-4710-beb9-434cfa1ed809-maas-api-xks-ca\") pod \"82c48696-42c4-4710-beb9-434cfa1ed809\" (UID: \"82c48696-42c4-4710-beb9-434cfa1ed809\") "
Nov 03 00:35:06.968040 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:06.957876    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp5n8\" (UniqueName: \"kubernetes.io/projected/93c536cd-5619-4f22-adb9-b7d1044815f0-kube-api-access-fp5n8\") pod \"maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz\" (UID: \"93c536cd-5619-4f22-adb9-b7d1044815f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz"
Nov 03 00:35:06.968040 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:06.957951    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c536cd-5619-4f22-adb9-b7d1044815f0-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz\" (UID: \"93c536cd-5619-4f22-adb9-b7d1044815f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz"
Nov 03 00:35:06.968040 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:06.958074    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c536cd-5619-4f22-adb9-b7d1044815f0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz\" (UID: \"93c536cd-5619-4f22-adb9-b7d1044815f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz"
Nov 03 00:35:06.968040 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:06.958134    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c536cd-5619-4f22-adb9-b7d1044815f0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz\" (UID: \"93c536cd-5619-4f22-adb9-b7d1044815f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz"
Nov 03 00:35:07.017890 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:07.017762    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82c48696-42c4-4710-beb9-434cfa1ed809-maas-api-openshift-ca" pod "82c48696-42c4-4710-beb9-434cfa1ed809" (UID: "82c48696-42c4-4710-beb9-434cfa1ed809"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:35:07.065156 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:07.065100    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82c48696-42c4-4710-beb9-434cfa1ed809-maas-api-xks-ca" pod "82c48696-42c4-4710-beb9-434cfa1ed809" (UID: "82c48696-42c4-4710-beb9-434cfa1ed809"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:35:07.115939 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:07.074780    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp5n8\" (UniqueName: \"kubernetes.io/projected/93c536cd-5619-4f22-adb9-b7d1044815f0-kube-api-access-fp5n8\") pod \"maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz\" (UID: \"93c536cd-5619-4f22-adb9-b7d1044815f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz"
Nov 03 00:35:07.319997 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:07.302956    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c536cd-5619-4f22-adb9-b7d1044815f0-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz\" (UID: \"93c536cd-5619-4f22-adb9-b7d1044815f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz"
Nov 03 00:35:07.321167 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:07.321132    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c536cd-5619-4f22-adb9-b7d1044815f0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz\" (UID: \"93c536cd-5619-4f22-adb9-b7d1044815f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz"
Nov 03 00:35:07.321423 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:07.321383    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c536cd-5619-4f22-adb9-b7d1044815f0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz\" (UID: \"93c536cd-5619-4f22-adb9-b7d1044815f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz"
Nov 03 00:35:07.321854 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:07.321833    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c48696-42c4-4710-beb9-434cfa1ed809-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:35:07.322040 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:07.322015    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c48696-42c4-4710-beb9-434cfa1ed809-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:35:07.322135 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:07.293301    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:07.322649 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:07.181713    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:07.323238 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:07.204180    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:07.392937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:07.085675    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82c48696-42c4-4710-beb9-434cfa1ed809-kube-api-access-8lmg7" pod "82c48696-42c4-4710-beb9-434cfa1ed809" (UID: "82c48696-42c4-4710-beb9-434cfa1ed809"). InnerVolumeSpecName "kube-api-access-8lmg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:35:07.392937 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:35:07.323216    5012 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-serving-cert: secret "maas-api-e2e-shared-a-w2-065de0-serving-cert" not found
Nov 03 00:35:07.392937 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:35:07.391182    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93c536cd-5619-4f22-adb9-b7d1044815f0-maas-api-tls podName:93c536cd-5619-4f22-adb9-b7d1044815f0 nodeName:}" failed. No retries permitted until 2026-11-03 00:35:07.891149707 +0000 UTC m=+1790.770733744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/93c536cd-5619-4f22-adb9-b7d1044815f0-maas-api-tls") pod "maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz" (UID: "93c536cd-5619-4f22-adb9-b7d1044815f0") : secret "maas-api-e2e-shared-a-w2-065de0-serving-cert" not found
Nov 03 00:35:07.392937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:07.391979    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp5n8\" (UniqueName: \"kubernetes.io/projected/93c536cd-5619-4f22-adb9-b7d1044815f0-kube-api-access-fp5n8\") pod \"maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz\" (UID: \"93c536cd-5619-4f22-adb9-b7d1044815f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz"
Nov 03 00:35:07.556751 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:07.556705    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c536cd-5619-4f22-adb9-b7d1044815f0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz\" (UID: \"93c536cd-5619-4f22-adb9-b7d1044815f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz"
Nov 03 00:35:07.607041 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:07.572445    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lmg7\" (UniqueName: \"kubernetes.io/projected/82c48696-42c4-4710-beb9-434cfa1ed809-kube-api-access-8lmg7\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:35:07.739017 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:07.738954    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded"
Nov 03 00:35:07.760865 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:07.760824    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c536cd-5619-4f22-adb9-b7d1044815f0-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz\" (UID: \"93c536cd-5619-4f22-adb9-b7d1044815f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz"
Nov 03 00:35:08.207644 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:08.207596    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 03 00:35:08.337511 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:08.323043    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:08.337511 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:08.323590    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h" podUID="21760fa1-e581-4657-b63e-d9dd3dde6084" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:08.337511 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:08.323839    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:08.455329 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:08.455276    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c536cd-5619-4f22-adb9-b7d1044815f0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz\" (UID: \"93c536cd-5619-4f22-adb9-b7d1044815f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz"
Nov 03 00:35:08.525968 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:35:08.525842    5012 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.284s"
Nov 03 00:35:08.615966 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:08.615868    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c536cd-5619-4f22-adb9-b7d1044815f0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz\" (UID: \"93c536cd-5619-4f22-adb9-b7d1044815f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz"
Nov 03 00:35:08.675055 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:08.670814    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 00:35:08.679658 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:08.679401    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d648ba4-9s9rs"
Nov 03 00:35:08.679658 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:08.679543    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d648ba4-9s9rs" event={"ID":"82c48696-42c4-4710-beb9-434cfa1ed809","Type":"ContainerDied","Data":"c64a7db2245cbeaacb87ef8a0849cf3925c514a8d3b45162a0f90285e4b93a7e"}
Nov 03 00:35:08.679658 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:08.679585    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c64a7db2245cbeaacb87ef8a0849cf3925c514a8d3b45162a0f90285e4b93a7e"
Nov 03 00:35:08.748931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:08.739471    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 00:35:08.761712 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:08.760865    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 00:35:08.761712 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:08.760964    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h"
Nov 03 00:35:08.779808 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:08.779531    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 00:35:08.823433 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:35:08.823389    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:35:08.900276 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:08.900240    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz"
Nov 03 00:35:09.245952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:09.241631    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz"]
Nov 03 00:35:09.285274 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:35:09.275030    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c536cd_5619_4f22_adb9_b7d1044815f0.slice/crio-4abfc1c7ca7217918538a627606e11e5f14b36e3782c32d3933b9525907ad596 WatchSource:0}: Error finding container 4abfc1c7ca7217918538a627606e11e5f14b36e3782c32d3933b9525907ad596: Status 404 returned error can't find the container with id 4abfc1c7ca7217918538a627606e11e5f14b36e3782c32d3933b9525907ad596
Nov 03 00:35:09.691758 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:09.691725    5012 generic.go:356] "Generic (PLEG): container finished" podID="6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5" containerID="8d1fb92ef148cf95773de678772db9c73a61acad6dff5adf202199fd22a23974" exitCode=0
Nov 03 00:35:09.692247 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:09.691799    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln" event={"ID":"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5","Type":"ContainerDied","Data":"8d1fb92ef148cf95773de678772db9c73a61acad6dff5adf202199fd22a23974"}
Nov 03 00:35:09.692247 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:09.691884    5012 scope.go:122] "RemoveContainer" containerID="45a709ea2c77440cc22cf716ce9e4b2d0eeefe0c48ed89ab9faa359d04fbffd3"
Nov 03 00:35:09.694206 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:09.694171    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz" event={"ID":"93c536cd-5619-4f22-adb9-b7d1044815f0","Type":"ContainerStarted","Data":"4abfc1c7ca7217918538a627606e11e5f14b36e3782c32d3933b9525907ad596"}
Nov 03 00:35:09.825925 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:09.825882    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:35:09.991996 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:09.991019    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-podinfo\") pod \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") "
Nov 03 00:35:09.991996 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:09.991159    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-kube-api-access-zfcjq\" (UniqueName: \"kubernetes.io/projected/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-kube-api-access-zfcjq\") pod \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") "
Nov 03 00:35:09.991996 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:09.991194    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-workload-socket\") pod \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") "
Nov 03 00:35:09.991996 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:09.991251    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-credential-socket\") pod \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") "
Nov 03 00:35:09.991996 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:09.991317    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-data\") pod \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") "
Nov 03 00:35:09.991996 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:09.991381    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istiod-ca-cert\") pod \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") "
Nov 03 00:35:09.991996 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:09.991437    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-workload-certs\") pod \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") "
Nov 03 00:35:09.991996 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:09.991464    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-token\" (UniqueName: \"kubernetes.io/projected/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-token\") pod \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") "
Nov 03 00:35:09.991996 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:09.991633    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-envoy\") pod \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\" (UID: \"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5\") "
Nov 03 00:35:10.010669 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:09.994015    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-data" pod "6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5" (UID: "6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:35:10.010669 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:09.994205    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-workload-socket" pod "6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5" (UID: "6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:35:10.010669 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:09.995017    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-workload-certs" pod "6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5" (UID: "6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:35:10.010669 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:09.996129    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istiod-ca-cert" pod "6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5" (UID: "6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:35:10.010669 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:09.996447    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-credential-socket" pod "6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5" (UID: "6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:35:10.011648 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:10.011448    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-token" pod "6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5" (UID: "6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:35:10.011648 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:10.011608    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-kube-api-access-zfcjq" pod "6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5" (UID: "6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5"). InnerVolumeSpecName "kube-api-access-zfcjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:35:10.012108 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:10.012067    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-envoy" pod "6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5" (UID: "6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:35:10.014469 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:10.014401    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-podinfo" pod "6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5" (UID: "6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:35:10.093840 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:10.093795    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:35:10.093840 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:10.093832    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfcjq\" (UniqueName: \"kubernetes.io/projected/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-kube-api-access-zfcjq\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:35:10.093840 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:10.093844    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:35:10.094143 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:10.093856    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:35:10.094143 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:10.093865    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:35:10.094143 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:10.093873    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:35:10.094143 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:10.093882    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:35:10.094143 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:10.093889    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:35:10.094143 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:10.093896    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:35:10.703597 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:10.703565    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"
Nov 03 00:35:10.704221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:10.703593    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln" event={"ID":"6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5","Type":"ContainerDied","Data":"f9de7a9a0a5e7b8946aeb67b69e1ae5a187d299de3ee9392d339aac565958673"}
Nov 03 00:35:10.704221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:10.703669    5012 scope.go:122] "RemoveContainer" containerID="8d1fb92ef148cf95773de678772db9c73a61acad6dff5adf202199fd22a23974"
Nov 03 00:35:10.705551 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:10.705523    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz" event={"ID":"93c536cd-5619-4f22-adb9-b7d1044815f0","Type":"ContainerStarted","Data":"9e094b0e024a4ad1849832cd7ac0700b0db4d73659879e35d92eaf927347c494"}
Nov 03 00:35:10.705652 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:10.705586    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz"
Nov 03 00:35:10.767156 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:10.767078    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz" podStartSLOduration=4.420609461 podStartE2EDuration="4.767061745s" totalImagesPullingTime="346.452284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:10.764949217 +0000 UTC m=+1793.644533257" watchObservedRunningTime="2026-11-03 00:35:10.767061745 +0000 UTC m=+1793.646645788"
Nov 03 00:35:10.807694 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:10.807642    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"]
Nov 03 00:35:10.817165 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:10.817122    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-a19e67-openshift-default-5f48fc9df7-bwxln"]
Nov 03 00:35:11.244666 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:11.244635    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5" path="/var/lib/kubelet/pods/6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5/volumes"
Nov 03 00:35:12.087822 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:12.087781    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd"]
Nov 03 00:35:12.088516 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:12.088498    5012 cpu_manager.go:414] "containerMap: removing container" podUID="6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5" containerName="istio-proxy"
Nov 03 00:35:12.088562 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:12.088517    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5" containerName="istio-proxy"
Nov 03 00:35:12.088562 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:12.088533    5012 cpu_manager.go:414] "containerMap: removing container" podUID="82c48696-42c4-4710-beb9-434cfa1ed809" containerName="revoke-keys"
Nov 03 00:35:12.088562 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:12.088538    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82c48696-42c4-4710-beb9-434cfa1ed809" containerName="revoke-keys"
Nov 03 00:35:12.088562 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:12.088554    5012 cpu_manager.go:414] "containerMap: removing container" podUID="6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5" containerName="istio-proxy"
Nov 03 00:35:12.088562 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:12.088560    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5" containerName="istio-proxy"
Nov 03 00:35:12.088729 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:12.088651    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5" containerName="istio-proxy"
Nov 03 00:35:12.088729 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:12.088658    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="82c48696-42c4-4710-beb9-434cfa1ed809" containerName="revoke-keys"
Nov 03 00:35:12.094024 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:12.093997    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd"
Nov 03 00:35:12.111948 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:12.111875    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd"]
Nov 03 00:35:12.116120 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:12.116084    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-e62935\""
Nov 03 00:35:12.116289 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:12.116207    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-e62935-dockercfg-nzlcd\""
Nov 03 00:35:12.263932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:12.262110    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w56g4\" (UniqueName: \"kubernetes.io/projected/5d0c60ba-1ab0-4a70-82b9-78580583e5cf-kube-api-access-w56g4\") pod \"payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd\" (UID: \"5d0c60ba-1ab0-4a70-82b9-78580583e5cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd"
Nov 03 00:35:12.263932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:12.262245    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d0c60ba-1ab0-4a70-82b9-78580583e5cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd\" (UID: \"5d0c60ba-1ab0-4a70-82b9-78580583e5cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd"
Nov 03 00:35:12.366022 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:12.365968    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w56g4\" (UniqueName: \"kubernetes.io/projected/5d0c60ba-1ab0-4a70-82b9-78580583e5cf-kube-api-access-w56g4\") pod \"payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd\" (UID: \"5d0c60ba-1ab0-4a70-82b9-78580583e5cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd"
Nov 03 00:35:12.366433 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:12.366401    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d0c60ba-1ab0-4a70-82b9-78580583e5cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd\" (UID: \"5d0c60ba-1ab0-4a70-82b9-78580583e5cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd"
Nov 03 00:35:12.368067 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:12.368037    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d0c60ba-1ab0-4a70-82b9-78580583e5cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd\" (UID: \"5d0c60ba-1ab0-4a70-82b9-78580583e5cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd"
Nov 03 00:35:12.396800 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:12.396767    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w56g4\" (UniqueName: \"kubernetes.io/projected/5d0c60ba-1ab0-4a70-82b9-78580583e5cf-kube-api-access-w56g4\") pod \"payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd\" (UID: \"5d0c60ba-1ab0-4a70-82b9-78580583e5cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd"
Nov 03 00:35:12.409635 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:12.409598    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd"
Nov 03 00:35:12.728466 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:12.728436    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd"]
Nov 03 00:35:12.729532 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:35:12.729504    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0c60ba_1ab0_4a70_82b9_78580583e5cf.slice/crio-18dd9584c6017173522ae4764a00b3ff8eef53db3aa1101cb71549a0ea09f08f WatchSource:0}: Error finding container 18dd9584c6017173522ae4764a00b3ff8eef53db3aa1101cb71549a0ea09f08f: Status 404 returned error can't find the container with id 18dd9584c6017173522ae4764a00b3ff8eef53db3aa1101cb71549a0ea09f08f
Nov 03 00:35:13.737480 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:13.737428    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd" event={"ID":"5d0c60ba-1ab0-4a70-82b9-78580583e5cf","Type":"ContainerStarted","Data":"119c0b800c3fa39ed88d4b22b3a430413422c2eb25665093cd221ef745d67a37"}
Nov 03 00:35:13.738109 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:13.737508    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd" event={"ID":"5d0c60ba-1ab0-4a70-82b9-78580583e5cf","Type":"ContainerStarted","Data":"18dd9584c6017173522ae4764a00b3ff8eef53db3aa1101cb71549a0ea09f08f"}
Nov 03 00:35:13.738109 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:13.738101    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd"
Nov 03 00:35:13.778357 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:13.778279    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd" podStartSLOduration=1.778260959 podStartE2EDuration="1.778260959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:13.773211059 +0000 UTC m=+1796.652795101" watchObservedRunningTime="2026-11-03 00:35:13.778260959 +0000 UTC m=+1796.657845052"
Nov 03 00:35:16.723412 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:16.723377    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz"
Nov 03 00:35:18.028269 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:18.028229    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 00:35:18.034617 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:18.034584    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:35:18.040125 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:18.040091    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 00:35:18.045798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:18.045769    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:35:21.045324 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.045288    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"]
Nov 03 00:35:21.046040 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.046023    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ddb91bd-5e27-43f9-bc78-e9c27ba02ac5" containerName="istio-proxy"
Nov 03 00:35:21.051554 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.051529    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.104878 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.104845    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"]
Nov 03 00:35:21.111583 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.111553    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"]
Nov 03 00:35:21.111804 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.111669    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.114816 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.114791    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"]
Nov 03 00:35:21.200884 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.200847    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.201104 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.200894    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.201104 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.200996    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.201104 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.201074    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.201104 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.201103    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f996bace-902b-4e8d-95bc-43f248835e98-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.201320 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.201121    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flv42\" (UniqueName: \"kubernetes.io/projected/a0ee7800-4f9e-4307-8043-3d10c5bde48b-kube-api-access-flv42\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.201320 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.201138    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.201320 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.201307    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.201452 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.201368    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.201489 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.201438    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-758rh\" (UniqueName: \"kubernetes.io/projected/f996bace-902b-4e8d-95bc-43f248835e98-kube-api-access-758rh\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.201489 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.201475    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.201607 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.201587    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.201685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.201671    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f996bace-902b-4e8d-95bc-43f248835e98-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.201744 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.201726    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.201821 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.201799    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.201861 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.201851    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f996bace-902b-4e8d-95bc-43f248835e98-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.201899 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.201891    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.202005 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.201990    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.302852 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.302768    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.302852 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.302809    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.302852 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.302842    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f996bace-902b-4e8d-95bc-43f248835e98-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.303229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.302894    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.303229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.302949    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.303229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.302997    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.303229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.303027    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.303229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.303046    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.303229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.303097    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.303229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.303125    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f996bace-902b-4e8d-95bc-43f248835e98-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.303229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.303143    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flv42\" (UniqueName: \"kubernetes.io/projected/a0ee7800-4f9e-4307-8043-3d10c5bde48b-kube-api-access-flv42\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.303229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.303160    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.303715 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.303232    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.303715 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.303291    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.303715 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.303336    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-758rh\" (UniqueName: \"kubernetes.io/projected/f996bace-902b-4e8d-95bc-43f248835e98-kube-api-access-758rh\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.303715 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.303363    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.303715 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.303503    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.303715 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.303514    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.303715 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.303556    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.303715 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.303590    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f996bace-902b-4e8d-95bc-43f248835e98-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.303715 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.303595    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.304232 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.303900    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.304232 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.304207    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.304344 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.304231    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.304956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.304592    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.304956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.304678    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.305163 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.305007    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f996bace-902b-4e8d-95bc-43f248835e98-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.305369 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.305340    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.305887 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.305841    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.306799 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.306773    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f996bace-902b-4e8d-95bc-43f248835e98-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.307142 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.307116    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.307320 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.307301    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.321075 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.321053    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f996bace-902b-4e8d-95bc-43f248835e98-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.321203 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.321181    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.321582 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.321566    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-758rh\" (UniqueName: \"kubernetes.io/projected/f996bace-902b-4e8d-95bc-43f248835e98-kube-api-access-758rh\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.321765 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.321748    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flv42\" (UniqueName: \"kubernetes.io/projected/a0ee7800-4f9e-4307-8043-3d10c5bde48b-kube-api-access-flv42\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.414599 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.414571    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:21.427151 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.427125    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:21.744942 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:35:21.741901    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf996bace_902b_4e8d_95bc_43f248835e98.slice/crio-955c8d6a3541a4d8a5fafba30b2bea42c35c714f4953ec216c6f56c0465681c3 WatchSource:0}: Error finding container 955c8d6a3541a4d8a5fafba30b2bea42c35c714f4953ec216c6f56c0465681c3: Status 404 returned error can't find the container with id 955c8d6a3541a4d8a5fafba30b2bea42c35c714f4953ec216c6f56c0465681c3
Nov 03 00:35:21.744942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.742820    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"]
Nov 03 00:35:21.757711 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.755520    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:21.757711 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.755619    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:21.757711 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.755663    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:21.791445 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.791397    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"]
Nov 03 00:35:21.798484 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.798439    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:21.798628 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.798520    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:21.798628 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.798556    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:21.909790 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.909758    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm" event={"ID":"f996bace-902b-4e8d-95bc-43f248835e98","Type":"ContainerStarted","Data":"edcd096742c694466ab096520881c14df0c07022639e0aa7965acc1dcccb12c4"}
Nov 03 00:35:21.910041 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.910022    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm" event={"ID":"f996bace-902b-4e8d-95bc-43f248835e98","Type":"ContainerStarted","Data":"955c8d6a3541a4d8a5fafba30b2bea42c35c714f4953ec216c6f56c0465681c3"}
Nov 03 00:35:21.916170 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.914000    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8" event={"ID":"a0ee7800-4f9e-4307-8043-3d10c5bde48b","Type":"ContainerStarted","Data":"1143bb2983498e9445cd2a2c11ea90f0243cbecb2de10b263227c74308949ced"}
Nov 03 00:35:21.916170 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.914078    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8" event={"ID":"a0ee7800-4f9e-4307-8043-3d10c5bde48b","Type":"ContainerStarted","Data":"7ee4c58dfa36da0ae55cb10c53075112f1d2443020f197d31ca2cb5b4c593928"}
Nov 03 00:35:21.992803 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:21.992607    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm" podStartSLOduration=0.992582374 podStartE2EDuration="992.582374ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:21.96848544 +0000 UTC m=+1804.848069484" watchObservedRunningTime="2026-11-03 00:35:21.992582374 +0000 UTC m=+1804.872166422"
Nov 03 00:35:22.010396 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:22.010323    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8" podStartSLOduration=1.01029986 podStartE2EDuration="1.01029986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:22.005797415 +0000 UTC m=+1804.885381462" watchObservedRunningTime="2026-11-03 00:35:22.01029986 +0000 UTC m=+1804.889883906"
Nov 03 00:35:22.415287 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:22.415247    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:22.416871 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:22.416838    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 03 00:35:22.417044 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:22.416906    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm" podUID="f996bace-902b-4e8d-95bc-43f248835e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 03 00:35:22.428106 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:22.428067    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:22.429804 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:22.429772    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 00:35:22.429970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:22.429841    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8" podUID="a0ee7800-4f9e-4307-8043-3d10c5bde48b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 00:35:23.417933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:23.416881    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 03 00:35:23.417933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:23.417008    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm" podUID="f996bace-902b-4e8d-95bc-43f248835e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 03 00:35:23.430968 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:23.430413    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 00:35:23.430968 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:23.430490    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8" podUID="a0ee7800-4f9e-4307-8043-3d10c5bde48b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 00:35:24.420935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:24.419891    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 03 00:35:24.420935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:24.419980    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm" podUID="f996bace-902b-4e8d-95bc-43f248835e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 03 00:35:24.443937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:24.440371    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 00:35:24.443937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:24.440437    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8" podUID="a0ee7800-4f9e-4307-8043-3d10c5bde48b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 00:35:24.752319 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:24.752141    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd"
Nov 03 00:35:25.419766 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:25.419721    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 03 00:35:25.420113 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:25.420078    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm" podUID="f996bace-902b-4e8d-95bc-43f248835e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 03 00:35:25.428309 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:25.428268    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 00:35:25.428984 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:25.428944    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8" podUID="a0ee7800-4f9e-4307-8043-3d10c5bde48b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 00:35:26.436933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:26.436094    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 00:35:26.436933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:26.436156    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8" podUID="a0ee7800-4f9e-4307-8043-3d10c5bde48b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 00:35:26.436933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:26.436599    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 03 00:35:26.436933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:26.436636    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm" podUID="f996bace-902b-4e8d-95bc-43f248835e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 03 00:35:26.840447 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:26.840354    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4"]
Nov 03 00:35:26.866956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:26.866675    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4"]
Nov 03 00:35:26.866956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:26.866780    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4"
Nov 03 00:35:26.881601 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:26.881579    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-c59124\""
Nov 03 00:35:26.908291 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:26.908263    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-c59124-dockercfg-hmq82\""
Nov 03 00:35:26.953272 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:26.953233    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/013f254b-241b-41e8-9974-56b75b830262-plugins-config-volume\") pod \"payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4\" (UID: \"013f254b-241b-41e8-9974-56b75b830262\") " pod="openshift-ingress/payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4"
Nov 03 00:35:26.953570 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:26.953546    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgwvj\" (UniqueName: \"kubernetes.io/projected/013f254b-241b-41e8-9974-56b75b830262-kube-api-access-wgwvj\") pod \"payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4\" (UID: \"013f254b-241b-41e8-9974-56b75b830262\") " pod="openshift-ingress/payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4"
Nov 03 00:35:27.055166 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:27.055127    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgwvj\" (UniqueName: \"kubernetes.io/projected/013f254b-241b-41e8-9974-56b75b830262-kube-api-access-wgwvj\") pod \"payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4\" (UID: \"013f254b-241b-41e8-9974-56b75b830262\") " pod="openshift-ingress/payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4"
Nov 03 00:35:27.055331 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:27.055300    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/013f254b-241b-41e8-9974-56b75b830262-plugins-config-volume\") pod \"payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4\" (UID: \"013f254b-241b-41e8-9974-56b75b830262\") " pod="openshift-ingress/payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4"
Nov 03 00:35:27.056027 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:27.056007    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/013f254b-241b-41e8-9974-56b75b830262-plugins-config-volume\") pod \"payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4\" (UID: \"013f254b-241b-41e8-9974-56b75b830262\") " pod="openshift-ingress/payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4"
Nov 03 00:35:27.073851 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:27.073807    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgwvj\" (UniqueName: \"kubernetes.io/projected/013f254b-241b-41e8-9974-56b75b830262-kube-api-access-wgwvj\") pod \"payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4\" (UID: \"013f254b-241b-41e8-9974-56b75b830262\") " pod="openshift-ingress/payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4"
Nov 03 00:35:27.186223 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:27.186177    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4"
Nov 03 00:35:27.443079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:27.415370    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 03 00:35:27.443079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:27.415437    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm" podUID="f996bace-902b-4e8d-95bc-43f248835e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 03 00:35:27.444180 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:27.444148    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 00:35:27.444406 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:27.444373    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8" podUID="a0ee7800-4f9e-4307-8043-3d10c5bde48b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 00:35:27.646590 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:27.646506    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4"]
Nov 03 00:35:28.043642 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:28.043513    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4" event={"ID":"013f254b-241b-41e8-9974-56b75b830262","Type":"ContainerStarted","Data":"6da04edb1ddcbf5687edc859e8fe5343a43a387aecde41dd47ff8505d518b083"}
Nov 03 00:35:28.043642 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:28.043578    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4" event={"ID":"013f254b-241b-41e8-9974-56b75b830262","Type":"ContainerStarted","Data":"e4287cc109e3338138196361993655604b8be08b8e36296d3b14c47321f6e7be"}
Nov 03 00:35:28.044080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:28.044062    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4"
Nov 03 00:35:28.084351 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:28.084282    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4" podStartSLOduration=2.084260888 podStartE2EDuration="2.084260888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:28.08067829 +0000 UTC m=+1810.960262333" watchObservedRunningTime="2026-11-03 00:35:28.084260888 +0000 UTC m=+1810.963844931"
Nov 03 00:35:28.426114 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:28.420341    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 03 00:35:28.426114 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:28.420407    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm" podUID="f996bace-902b-4e8d-95bc-43f248835e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 03 00:35:28.439971 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:28.433378    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 00:35:28.439971 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:28.433443    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8" podUID="a0ee7800-4f9e-4307-8043-3d10c5bde48b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 00:35:29.415160 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:29.415118    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 03 00:35:29.415697 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:29.415190    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm" podUID="f996bace-902b-4e8d-95bc-43f248835e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 03 00:35:29.428585 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:29.428536    5012 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 00:35:29.428761 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:29.428610    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8" podUID="a0ee7800-4f9e-4307-8043-3d10c5bde48b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 00:35:30.767944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:30.767888    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:30.769626 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:30.769518    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:30.780546 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:30.780467    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:35:30.847650 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:30.847604    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:30.848811 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:30.848779    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:30.850060 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:30.850036    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:35:33.074742 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.074701    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"]
Nov 03 00:35:33.082066 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.082034    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.114608 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.114572    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-7ef1b6-openshift-default-dockercfg-b2xjx\""
Nov 03 00:35:33.129763 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.129727    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"]
Nov 03 00:35:33.304321 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.304258    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-workload-certs\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.304642 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.304621    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-token\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.304809 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.304793    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-envoy\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.305125 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.305098    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-workload-socket\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.305336 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.305316    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4tm9\" (UniqueName: \"kubernetes.io/projected/dbbcfc46-aa01-443c-a475-a757649f86ce-kube-api-access-f4tm9\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.305520 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.305505    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-podinfo\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.305684 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.305669    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-data\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.306006 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.305982    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-credential-socket\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.306194 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.306177    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbbcfc46-aa01-443c-a475-a757649f86ce-istiod-ca-cert\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.459990 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.459950    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4tm9\" (UniqueName: \"kubernetes.io/projected/dbbcfc46-aa01-443c-a475-a757649f86ce-kube-api-access-f4tm9\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.460185 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.460031    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-podinfo\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.460185 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.460072    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-data\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.460185 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.460090    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-credential-socket\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.460185 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.460130    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbbcfc46-aa01-443c-a475-a757649f86ce-istiod-ca-cert\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.460359 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.460232    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-workload-certs\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.460359 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.460252    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-token\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.460359 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.460287    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-envoy\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.460359 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.460310    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-workload-socket\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.460865 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.460831    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-data\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.460986 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.460929    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-workload-certs\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.460986 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.460955    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-credential-socket\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.461121 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.461023    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-workload-socket\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.461265 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.461237    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbbcfc46-aa01-443c-a475-a757649f86ce-istiod-ca-cert\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.466079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.466047    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-podinfo\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.466335 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.466313    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-envoy\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.510503 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.510470    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-token\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.510780 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.510756    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4tm9\" (UniqueName: \"kubernetes.io/projected/dbbcfc46-aa01-443c-a475-a757649f86ce-kube-api-access-f4tm9\") pod \"e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") " pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:33.705716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:33.705671    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:34.187450 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:34.187396    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"]
Nov 03 00:35:34.189212 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:35:34.189176    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbbcfc46_aa01_443c_a475_a757649f86ce.slice/crio-a6867856ef617c362916988988eec8614b746fe8e29a4040e40badc2c30af7be WatchSource:0}: Error finding container a6867856ef617c362916988988eec8614b746fe8e29a4040e40badc2c30af7be: Status 404 returned error can't find the container with id a6867856ef617c362916988988eec8614b746fe8e29a4040e40badc2c30af7be
Nov 03 00:35:34.192799 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:34.192743    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:34.193047 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:34.192854    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:34.193047 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:34.192933    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:35.149037 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:35.148992    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd" event={"ID":"dbbcfc46-aa01-443c-a475-a757649f86ce","Type":"ContainerStarted","Data":"bc2f9bac5940460d145b142b868b1dcd75e0d4c5bdd5f1542e0d841235297be2"}
Nov 03 00:35:35.149315 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:35.149074    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd" event={"ID":"dbbcfc46-aa01-443c-a475-a757649f86ce","Type":"ContainerStarted","Data":"a6867856ef617c362916988988eec8614b746fe8e29a4040e40badc2c30af7be"}
Nov 03 00:35:35.718968 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:35.713036    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:35.815044 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:35.814338    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:35.864944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:35.864876    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd" podStartSLOduration=2.864863102 podStartE2EDuration="2.864863102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:35.206779319 +0000 UTC m=+1818.086363362" watchObservedRunningTime="2026-11-03 00:35:35.864863102 +0000 UTC m=+1818.744447145"
Nov 03 00:35:35.949295 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:35.949253    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"]
Nov 03 00:35:35.949530 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:35.949502    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc" podUID="51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" containerName="istio-proxy" containerID="cri-o://5439684e89b69f602bda859029a342e8564bf1754cf650752edcf7eebe8f9730" gracePeriod=30
Nov 03 00:35:36.157001 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:36.156965    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:36.158115 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:36.158093    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:35:37.364341 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:35:37.364302    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:35:39.054942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:39.054881    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4"
Nov 03 00:35:39.674850 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:39.674806    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn"]
Nov 03 00:35:39.679276 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:39.679259    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn"
Nov 03 00:35:39.694549 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:39.694524    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-7ef1b6\""
Nov 03 00:35:39.697174 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:39.697144    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn"]
Nov 03 00:35:39.773700 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:39.773653    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9jfj\" (UniqueName: \"kubernetes.io/projected/36585f09-b213-48d7-b992-e0f2811a7072-kube-api-access-s9jfj\") pod \"payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn\" (UID: \"36585f09-b213-48d7-b992-e0f2811a7072\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn"
Nov 03 00:35:39.773943 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:39.773920    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36585f09-b213-48d7-b992-e0f2811a7072-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn\" (UID: \"36585f09-b213-48d7-b992-e0f2811a7072\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn"
Nov 03 00:35:39.915177 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:39.912224    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9jfj\" (UniqueName: \"kubernetes.io/projected/36585f09-b213-48d7-b992-e0f2811a7072-kube-api-access-s9jfj\") pod \"payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn\" (UID: \"36585f09-b213-48d7-b992-e0f2811a7072\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn"
Nov 03 00:35:39.915177 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:39.912555    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36585f09-b213-48d7-b992-e0f2811a7072-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn\" (UID: \"36585f09-b213-48d7-b992-e0f2811a7072\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn"
Nov 03 00:35:39.995986 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:39.990357    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36585f09-b213-48d7-b992-e0f2811a7072-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn\" (UID: \"36585f09-b213-48d7-b992-e0f2811a7072\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn"
Nov 03 00:35:40.024830 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:40.024788    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9jfj\" (UniqueName: \"kubernetes.io/projected/36585f09-b213-48d7-b992-e0f2811a7072-kube-api-access-s9jfj\") pod \"payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn\" (UID: \"36585f09-b213-48d7-b992-e0f2811a7072\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn"
Nov 03 00:35:40.295095 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:40.295007    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn"
Nov 03 00:35:40.602039 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:35:40.602001    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36585f09_b213_48d7_b992_e0f2811a7072.slice/crio-c6f7dfead6387e4ad8fc41f1bf80f1aa0740ee93ef0922c5cff044b256377801 WatchSource:0}: Error finding container c6f7dfead6387e4ad8fc41f1bf80f1aa0740ee93ef0922c5cff044b256377801: Status 404 returned error can't find the container with id c6f7dfead6387e4ad8fc41f1bf80f1aa0740ee93ef0922c5cff044b256377801
Nov 03 00:35:40.602193 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:40.602161    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn"]
Nov 03 00:35:41.231308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.231250    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:35:41.281643 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.272363    5012 generic.go:356] "Generic (PLEG): container finished" podID="51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" containerID="5439684e89b69f602bda859029a342e8564bf1754cf650752edcf7eebe8f9730" exitCode=0
Nov 03 00:35:41.281643 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.272528    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"
Nov 03 00:35:41.289142 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.289102    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc" event={"ID":"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c","Type":"ContainerDied","Data":"5439684e89b69f602bda859029a342e8564bf1754cf650752edcf7eebe8f9730"}
Nov 03 00:35:41.289335 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.289154    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc" event={"ID":"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c","Type":"ContainerDied","Data":"21fff050a27d4a3646ad0e42b34501d514ae757ccf9322b53a1457bb1bfb12d6"}
Nov 03 00:35:41.289335 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.289179    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn" event={"ID":"36585f09-b213-48d7-b992-e0f2811a7072","Type":"ContainerStarted","Data":"5478b7ab1b64c3a861b34b07ac136b5e31ea526f37be8b4f025cd51a17883d21"}
Nov 03 00:35:41.289335 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.289215    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn"
Nov 03 00:35:41.289335 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.289236    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn" event={"ID":"36585f09-b213-48d7-b992-e0f2811a7072","Type":"ContainerStarted","Data":"c6f7dfead6387e4ad8fc41f1bf80f1aa0740ee93ef0922c5cff044b256377801"}
Nov 03 00:35:41.290819 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.290262    5012 scope.go:122] "RemoveContainer" containerID="5439684e89b69f602bda859029a342e8564bf1754cf650752edcf7eebe8f9730"
Nov 03 00:35:41.311972 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.311944    5012 scope.go:122] "RemoveContainer" containerID="18a8893cac47b63fca1c159aea7507e8c7a080459cbadd0005de6e25326804d6"
Nov 03 00:35:41.323092 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.323051    5012 scope.go:122] "RemoveContainer" containerID="5439684e89b69f602bda859029a342e8564bf1754cf650752edcf7eebe8f9730"
Nov 03 00:35:41.323436 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:35:41.323411    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5439684e89b69f602bda859029a342e8564bf1754cf650752edcf7eebe8f9730\": container with ID starting with 5439684e89b69f602bda859029a342e8564bf1754cf650752edcf7eebe8f9730 not found: ID does not exist" containerID="5439684e89b69f602bda859029a342e8564bf1754cf650752edcf7eebe8f9730"
Nov 03 00:35:41.323530 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.323449    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5439684e89b69f602bda859029a342e8564bf1754cf650752edcf7eebe8f9730"} err="failed to get container status \"5439684e89b69f602bda859029a342e8564bf1754cf650752edcf7eebe8f9730\": rpc error: code = NotFound desc = could not find container \"5439684e89b69f602bda859029a342e8564bf1754cf650752edcf7eebe8f9730\": container with ID starting with 5439684e89b69f602bda859029a342e8564bf1754cf650752edcf7eebe8f9730 not found: ID does not exist"
Nov 03 00:35:41.323530 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.323472    5012 scope.go:122] "RemoveContainer" containerID="18a8893cac47b63fca1c159aea7507e8c7a080459cbadd0005de6e25326804d6"
Nov 03 00:35:41.323860 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:35:41.323829    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18a8893cac47b63fca1c159aea7507e8c7a080459cbadd0005de6e25326804d6\": container with ID starting with 18a8893cac47b63fca1c159aea7507e8c7a080459cbadd0005de6e25326804d6 not found: ID does not exist" containerID="18a8893cac47b63fca1c159aea7507e8c7a080459cbadd0005de6e25326804d6"
Nov 03 00:35:41.324078 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.323869    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18a8893cac47b63fca1c159aea7507e8c7a080459cbadd0005de6e25326804d6"} err="failed to get container status \"18a8893cac47b63fca1c159aea7507e8c7a080459cbadd0005de6e25326804d6\": rpc error: code = NotFound desc = could not find container \"18a8893cac47b63fca1c159aea7507e8c7a080459cbadd0005de6e25326804d6\": container with ID starting with 18a8893cac47b63fca1c159aea7507e8c7a080459cbadd0005de6e25326804d6 not found: ID does not exist"
Nov 03 00:35:41.333989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.333470    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-credential-socket\") pod \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") "
Nov 03 00:35:41.333989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.333555    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-podinfo\") pod \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") "
Nov 03 00:35:41.333989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.333646    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-data\") pod \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") "
Nov 03 00:35:41.333989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.333674    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-envoy\") pod \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") "
Nov 03 00:35:41.333989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.333706    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-kube-api-access-vv2h7\" (UniqueName: \"kubernetes.io/projected/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-kube-api-access-vv2h7\") pod \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") "
Nov 03 00:35:41.333989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.333726    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-credential-socket" pod "51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" (UID: "51a1f17e-efc6-4c72-ad9d-9b63ee81b92c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:35:41.333989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.333742    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-workload-certs\") pod \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") "
Nov 03 00:35:41.333989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.333844    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istiod-ca-cert\") pod \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") "
Nov 03 00:35:41.333989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.333871    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-token\" (UniqueName: \"kubernetes.io/projected/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-token\") pod \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") "
Nov 03 00:35:41.333989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.333986    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-workload-socket\") pod \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\" (UID: \"51a1f17e-efc6-4c72-ad9d-9b63ee81b92c\") "
Nov 03 00:35:41.334697 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.334175    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istiod-ca-cert" pod "51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" (UID: "51a1f17e-efc6-4c72-ad9d-9b63ee81b92c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:35:41.334697 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.334450    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-workload-certs" pod "51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" (UID: "51a1f17e-efc6-4c72-ad9d-9b63ee81b92c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:35:41.334829 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.334789    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:35:41.334829 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.334812    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:35:41.334829 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.334827    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:35:41.335352 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.335050    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-data" pod "51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" (UID: "51a1f17e-efc6-4c72-ad9d-9b63ee81b92c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:35:41.335641 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.335608    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-workload-socket" pod "51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" (UID: "51a1f17e-efc6-4c72-ad9d-9b63ee81b92c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:35:41.337997 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.337973    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-kube-api-access-vv2h7" pod "51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" (UID: "51a1f17e-efc6-4c72-ad9d-9b63ee81b92c"). InnerVolumeSpecName "kube-api-access-vv2h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:35:41.338084 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.338010    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-token" pod "51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" (UID: "51a1f17e-efc6-4c72-ad9d-9b63ee81b92c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:35:41.338175 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.338150    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-podinfo" pod "51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" (UID: "51a1f17e-efc6-4c72-ad9d-9b63ee81b92c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:35:41.343282 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.343237    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-envoy" pod "51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" (UID: "51a1f17e-efc6-4c72-ad9d-9b63ee81b92c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:35:41.347427 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.347373    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn" podStartSLOduration=2.347356444 podStartE2EDuration="2.347356444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:41.343704695 +0000 UTC m=+1824.223288738" watchObservedRunningTime="2026-11-03 00:35:41.347356444 +0000 UTC m=+1824.226940488"
Nov 03 00:35:41.436485 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.436442    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:35:41.436485 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.436479    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:35:41.436485 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.436494    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv2h7\" (UniqueName: \"kubernetes.io/projected/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-kube-api-access-vv2h7\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:35:41.436756 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.436506    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:35:41.436756 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.436520    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:35:41.436756 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.436531    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:35:41.639501 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.639464    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"]
Nov 03 00:35:41.649867 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:41.649840    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8fzrc"]
Nov 03 00:35:43.124397 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:43.124357    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6"]
Nov 03 00:35:43.125095 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:43.125078    5012 cpu_manager.go:414] "containerMap: removing container" podUID="51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" containerName="istio-proxy"
Nov 03 00:35:43.125095 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:43.125095    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" containerName="istio-proxy"
Nov 03 00:35:43.125229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:43.125110    5012 cpu_manager.go:414] "containerMap: removing container" podUID="51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" containerName="istio-proxy"
Nov 03 00:35:43.125229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:43.125115    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" containerName="istio-proxy"
Nov 03 00:35:43.125229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:43.125196    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" containerName="istio-proxy"
Nov 03 00:35:43.125229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:43.125206    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" containerName="istio-proxy"
Nov 03 00:35:43.131028 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:43.131001    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6"
Nov 03 00:35:43.147663 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:43.147639    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6"]
Nov 03 00:35:43.152999 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:43.152971    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-c59124-serving-cert\""
Nov 03 00:35:43.245371 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:43.245339    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51a1f17e-efc6-4c72-ad9d-9b63ee81b92c" path="/var/lib/kubelet/pods/51a1f17e-efc6-4c72-ad9d-9b63ee81b92c/volumes"
Nov 03 00:35:43.267591 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:43.267562    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5bnr\" (UniqueName: \"kubernetes.io/projected/589bae50-3c0e-466e-849b-cd6052804d65-kube-api-access-w5bnr\") pod \"maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6\" (UID: \"589bae50-3c0e-466e-849b-cd6052804d65\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6"
Nov 03 00:35:43.267801 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:43.267765    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/589bae50-3c0e-466e-849b-cd6052804d65-openshift-service-ca\") pod \"maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6\" (UID: \"589bae50-3c0e-466e-849b-cd6052804d65\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6"
Nov 03 00:35:43.267872 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:43.267803    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/589bae50-3c0e-466e-849b-cd6052804d65-maas-api-tls\") pod \"maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6\" (UID: \"589bae50-3c0e-466e-849b-cd6052804d65\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6"
Nov 03 00:35:43.268004 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:43.267984    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/589bae50-3c0e-466e-849b-cd6052804d65-metrics-certs\") pod \"maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6\" (UID: \"589bae50-3c0e-466e-849b-cd6052804d65\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6"
Nov 03 00:35:43.376630 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:43.376513    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/589bae50-3c0e-466e-849b-cd6052804d65-openshift-service-ca\") pod \"maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6\" (UID: \"589bae50-3c0e-466e-849b-cd6052804d65\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6"
Nov 03 00:35:43.377768 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:43.377724    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/589bae50-3c0e-466e-849b-cd6052804d65-openshift-service-ca\") pod \"maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6\" (UID: \"589bae50-3c0e-466e-849b-cd6052804d65\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6"
Nov 03 00:35:43.377973 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:43.377948    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/589bae50-3c0e-466e-849b-cd6052804d65-maas-api-tls\") pod \"maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6\" (UID: \"589bae50-3c0e-466e-849b-cd6052804d65\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6"
Nov 03 00:35:43.378193 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:43.378179    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/589bae50-3c0e-466e-849b-cd6052804d65-metrics-certs\") pod \"maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6\" (UID: \"589bae50-3c0e-466e-849b-cd6052804d65\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6"
Nov 03 00:35:43.378352 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:43.378339    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5bnr\" (UniqueName: \"kubernetes.io/projected/589bae50-3c0e-466e-849b-cd6052804d65-kube-api-access-w5bnr\") pod \"maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6\" (UID: \"589bae50-3c0e-466e-849b-cd6052804d65\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6"
Nov 03 00:35:43.383376 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:43.382837    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/589bae50-3c0e-466e-849b-cd6052804d65-maas-api-tls\") pod \"maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6\" (UID: \"589bae50-3c0e-466e-849b-cd6052804d65\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6"
Nov 03 00:35:43.383376 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:43.383312    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/589bae50-3c0e-466e-849b-cd6052804d65-metrics-certs\") pod \"maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6\" (UID: \"589bae50-3c0e-466e-849b-cd6052804d65\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6"
Nov 03 00:35:43.398393 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:43.398364    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5bnr\" (UniqueName: \"kubernetes.io/projected/589bae50-3c0e-466e-849b-cd6052804d65-kube-api-access-w5bnr\") pod \"maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6\" (UID: \"589bae50-3c0e-466e-849b-cd6052804d65\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6"
Nov 03 00:35:43.445739 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:43.445703    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6"
Nov 03 00:35:43.752562 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:43.752534    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6"]
Nov 03 00:35:43.752754 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:35:43.752734    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod589bae50_3c0e_466e_849b_cd6052804d65.slice/crio-67adb3eefcba6c93ac6b6e8c0a157019c8dd839d1fa523b669da724b7708c193 WatchSource:0}: Error finding container 67adb3eefcba6c93ac6b6e8c0a157019c8dd839d1fa523b669da724b7708c193: Status 404 returned error can't find the container with id 67adb3eefcba6c93ac6b6e8c0a157019c8dd839d1fa523b669da724b7708c193
Nov 03 00:35:44.302329 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:44.302290    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6" event={"ID":"589bae50-3c0e-466e-849b-cd6052804d65","Type":"ContainerStarted","Data":"fc853407f2a11c3ce5ea4a56868afd0448b2fff90c9b8c537157349dfb1165bf"}
Nov 03 00:35:44.302730 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:44.302352    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6" event={"ID":"589bae50-3c0e-466e-849b-cd6052804d65","Type":"ContainerStarted","Data":"67adb3eefcba6c93ac6b6e8c0a157019c8dd839d1fa523b669da724b7708c193"}
Nov 03 00:35:44.302730 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:44.302383    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6"
Nov 03 00:35:44.340697 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:44.340637    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6" podStartSLOduration=0.98994809 podStartE2EDuration="1.340624268s" totalImagesPullingTime="350.676178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:44.339768428 +0000 UTC m=+1827.219352472" watchObservedRunningTime="2026-11-03 00:35:44.340624268 +0000 UTC m=+1827.220208310"
Nov 03 00:35:47.771849 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:47.771792    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:50.318250 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:50.318217    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6"
Nov 03 00:35:51.909034 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:51.908997    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"]
Nov 03 00:35:51.938481 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:51.938444    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:51.974932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:51.958998    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"]
Nov 03 00:35:51.977109 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:51.977069    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-workload-certs\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:51.977483 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:51.977462    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/209af59a-bfd0-4aab-b2bf-7c789190f57e-istiod-ca-cert\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:51.977704 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:51.977686    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-credential-socket\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:51.977887 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:51.977866    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-workload-socket\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:51.978151 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:51.978131    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-data\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:51.978309 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:51.978292    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-token\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:51.982049 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:51.982022    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89wxp\" (UniqueName: \"kubernetes.io/projected/209af59a-bfd0-4aab-b2bf-7c789190f57e-kube-api-access-89wxp\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:51.982285 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:51.982266    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-envoy\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:52.010075 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:51.994800    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-podinfo\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:52.100331 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:52.079539    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-a07a90-openshift-default-dockercfg-ttwzx\""
Nov 03 00:35:52.219036 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:52.218894    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-credential-socket\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:52.219350 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:52.219324    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-workload-socket\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:52.219513 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:52.219496    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-data\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:52.220000 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:52.219980    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-token\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:52.220501 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:52.220478    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89wxp\" (UniqueName: \"kubernetes.io/projected/209af59a-bfd0-4aab-b2bf-7c789190f57e-kube-api-access-89wxp\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:52.220662 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:52.220646    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-envoy\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:52.220783 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:52.220766    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-podinfo\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:52.221083 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:52.221063    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-workload-certs\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:52.222330 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:52.222299    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/209af59a-bfd0-4aab-b2bf-7c789190f57e-istiod-ca-cert\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:52.222431 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:52.219663    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-credential-socket\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:52.222516 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:52.219934    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-data\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:52.222516 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:52.219628    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-workload-socket\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:52.222516 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:52.221436    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-workload-certs\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:52.223212 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:52.223185    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/209af59a-bfd0-4aab-b2bf-7c789190f57e-istiod-ca-cert\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:52.224186 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:52.224166    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-envoy\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:52.224999 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:52.224981    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-podinfo\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:52.243521 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:52.243496    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-token\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:52.249697 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:52.249675    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89wxp\" (UniqueName: \"kubernetes.io/projected/209af59a-bfd0-4aab-b2bf-7c789190f57e-kube-api-access-89wxp\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:52.318102 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:52.318063    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn"
Nov 03 00:35:52.464985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:52.464954    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:52.886684 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:35:52.886654    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209af59a_bfd0_4aab_b2bf_7c789190f57e.slice/crio-eab23cef46938d1b91b554b57d31912332b7d8ff7db30f52d93b1a092a832694 WatchSource:0}: Error finding container eab23cef46938d1b91b554b57d31912332b7d8ff7db30f52d93b1a092a832694: Status 404 returned error can't find the container with id eab23cef46938d1b91b554b57d31912332b7d8ff7db30f52d93b1a092a832694
Nov 03 00:35:52.887160 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:52.887058    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"]
Nov 03 00:35:52.889956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:52.889894    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:52.890059 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:52.890016    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:52.890112 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:52.890079    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:53.471145 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:53.471102    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k" event={"ID":"209af59a-bfd0-4aab-b2bf-7c789190f57e","Type":"ContainerStarted","Data":"5c0755ea0d94d72ad025aa98c515eb99c1a341bd0a7db744155a383853f2ecf8"}
Nov 03 00:35:53.471574 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:53.471172    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k" event={"ID":"209af59a-bfd0-4aab-b2bf-7c789190f57e","Type":"ContainerStarted","Data":"eab23cef46938d1b91b554b57d31912332b7d8ff7db30f52d93b1a092a832694"}
Nov 03 00:35:53.522142 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:53.522080    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k" podStartSLOduration=2.522062585 podStartE2EDuration="2.522062585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:53.520835522 +0000 UTC m=+1836.400419566" watchObservedRunningTime="2026-11-03 00:35:53.522062585 +0000 UTC m=+1836.401646629"
Nov 03 00:35:54.465221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:54.465171    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:54.471444 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:54.471411    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:54.479921 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:54.479883    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:54.480862 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:54.480838    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:35:56.532599 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:56.532561    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt"]
Nov 03 00:35:56.539469 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:56.539440    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt"
Nov 03 00:35:56.560725 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:56.560695    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt"]
Nov 03 00:35:56.561138 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:56.561109    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-9908e8-serving-cert\""
Nov 03 00:35:56.568053 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:56.568021    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt\" (UID: \"98a31bfd-3a54-4fac-b1ff-d768f6ec5504\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt"
Nov 03 00:35:56.568198 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:56.568107    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcq4v\" (UniqueName: \"kubernetes.io/projected/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-kube-api-access-rcq4v\") pod \"maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt\" (UID: \"98a31bfd-3a54-4fac-b1ff-d768f6ec5504\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt"
Nov 03 00:35:56.568198 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:56.568164    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt\" (UID: \"98a31bfd-3a54-4fac-b1ff-d768f6ec5504\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt"
Nov 03 00:35:56.568327 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:56.568245    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt\" (UID: \"98a31bfd-3a54-4fac-b1ff-d768f6ec5504\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt"
Nov 03 00:35:56.670228 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:56.670193    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt\" (UID: \"98a31bfd-3a54-4fac-b1ff-d768f6ec5504\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt"
Nov 03 00:35:56.670414 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:56.670241    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcq4v\" (UniqueName: \"kubernetes.io/projected/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-kube-api-access-rcq4v\") pod \"maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt\" (UID: \"98a31bfd-3a54-4fac-b1ff-d768f6ec5504\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt"
Nov 03 00:35:56.670414 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:56.670276    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt\" (UID: \"98a31bfd-3a54-4fac-b1ff-d768f6ec5504\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt"
Nov 03 00:35:56.670414 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:56.670325    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt\" (UID: \"98a31bfd-3a54-4fac-b1ff-d768f6ec5504\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt"
Nov 03 00:35:56.670632 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:35:56.670443    5012 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-serving-cert: secret "maas-api-e2e-shared-a-w4-9908e8-serving-cert" not found
Nov 03 00:35:56.670632 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:35:56.670514    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-maas-api-tls podName:98a31bfd-3a54-4fac-b1ff-d768f6ec5504 nodeName:}" failed. No retries permitted until 2026-11-03 00:35:57.17049514 +0000 UTC m=+1840.050079166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-maas-api-tls") pod "maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt" (UID: "98a31bfd-3a54-4fac-b1ff-d768f6ec5504") : secret "maas-api-e2e-shared-a-w4-9908e8-serving-cert" not found
Nov 03 00:35:56.671028 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:56.671010    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt\" (UID: \"98a31bfd-3a54-4fac-b1ff-d768f6ec5504\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt"
Nov 03 00:35:56.672754 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:56.672732    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt\" (UID: \"98a31bfd-3a54-4fac-b1ff-d768f6ec5504\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt"
Nov 03 00:35:56.690985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:56.690964    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcq4v\" (UniqueName: \"kubernetes.io/projected/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-kube-api-access-rcq4v\") pod \"maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt\" (UID: \"98a31bfd-3a54-4fac-b1ff-d768f6ec5504\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt"
Nov 03 00:35:57.194804 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:57.194738    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt\" (UID: \"98a31bfd-3a54-4fac-b1ff-d768f6ec5504\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt"
Nov 03 00:35:57.197096 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:57.197067    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt\" (UID: \"98a31bfd-3a54-4fac-b1ff-d768f6ec5504\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt"
Nov 03 00:35:57.457133 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:57.457028    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt"
Nov 03 00:35:57.516808 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:57.516776    5012 generic.go:356] "Generic (PLEG): container finished" podID="5795e973-761d-43af-91ea-0451024c0d42" containerID="793b72e6c7c6c379be340ef4e6637111ace89e4ca35f2443ed4fcaf265e01488" exitCode=0
Nov 03 00:35:57.517034 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:57.516859    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94" event={"ID":"5795e973-761d-43af-91ea-0451024c0d42","Type":"ContainerDied","Data":"793b72e6c7c6c379be340ef4e6637111ace89e4ca35f2443ed4fcaf265e01488"}
Nov 03 00:35:57.517470 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:57.517449    5012 scope.go:122] "RemoveContainer" containerID="793b72e6c7c6c379be340ef4e6637111ace89e4ca35f2443ed4fcaf265e01488"
Nov 03 00:35:57.519822 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:57.519784    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:57.519989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:57.519888    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:57.520284 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:57.519983    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:57.770048 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:57.770022    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt"]
Nov 03 00:35:57.771540 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:35:57.771513    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a31bfd_3a54_4fac_b1ff_d768f6ec5504.slice/crio-84e24714688092a9da9179de2241fe3fe28cbab2879212f48bb0d6c50b6978bc WatchSource:0}: Error finding container 84e24714688092a9da9179de2241fe3fe28cbab2879212f48bb0d6c50b6978bc: Status 404 returned error can't find the container with id 84e24714688092a9da9179de2241fe3fe28cbab2879212f48bb0d6c50b6978bc
Nov 03 00:35:58.528804 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:58.528721    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94" event={"ID":"5795e973-761d-43af-91ea-0451024c0d42","Type":"ContainerStarted","Data":"fee4d8bff4b21c9b375b12109683697675d5a69deda156a04e6d759b3c54faab"}
Nov 03 00:35:58.530716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:58.530673    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt" event={"ID":"98a31bfd-3a54-4fac-b1ff-d768f6ec5504","Type":"ContainerStarted","Data":"a5e5ae4d3c0edc04fdc4d3ee216517b5c8a7896ec27765289394db5d777ddb1f"}
Nov 03 00:35:58.530981 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:58.530957    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt"
Nov 03 00:35:58.531087 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:58.530993    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt" event={"ID":"98a31bfd-3a54-4fac-b1ff-d768f6ec5504","Type":"ContainerStarted","Data":"84e24714688092a9da9179de2241fe3fe28cbab2879212f48bb0d6c50b6978bc"}
Nov 03 00:35:58.625896 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:58.625839    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt" podStartSLOduration=2.26176707 podStartE2EDuration="2.625825548s" totalImagesPullingTime="364.058478ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:58.622638113 +0000 UTC m=+1841.502222155" watchObservedRunningTime="2026-11-03 00:35:58.625825548 +0000 UTC m=+1841.505409590"
Nov 03 00:35:58.948433 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:58.948398    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:35:58.948978 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:58.948531    5012 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 03 00:35:58.948978 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:58.948571    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94" podUID="5795e973-761d-43af-91ea-0451024c0d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 03 00:35:59.946658 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:59.946622    5012 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 03 00:35:59.946850 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:35:59.946677    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94" podUID="5795e973-761d-43af-91ea-0451024c0d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 03 00:36:00.416552 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:00.416516    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899"]
Nov 03 00:36:00.421323 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:00.421303    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899"
Nov 03 00:36:00.430028 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:00.429997    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0136143-9e52-4e16-aa8a-0e190e0239e2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899\" (UID: \"c0136143-9e52-4e16-aa8a-0e190e0239e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899"
Nov 03 00:36:00.430138 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:00.430043    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7686\" (UniqueName: \"kubernetes.io/projected/c0136143-9e52-4e16-aa8a-0e190e0239e2-kube-api-access-c7686\") pod \"maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899\" (UID: \"c0136143-9e52-4e16-aa8a-0e190e0239e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899"
Nov 03 00:36:00.430196 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:00.430168    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0136143-9e52-4e16-aa8a-0e190e0239e2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899\" (UID: \"c0136143-9e52-4e16-aa8a-0e190e0239e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899"
Nov 03 00:36:00.430465 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:00.430446    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0136143-9e52-4e16-aa8a-0e190e0239e2-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899\" (UID: \"c0136143-9e52-4e16-aa8a-0e190e0239e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899"
Nov 03 00:36:00.437598 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:00.437573    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899"]
Nov 03 00:36:00.442098 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:00.442075    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-7ef1b6-serving-cert\""
Nov 03 00:36:00.531614 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:00.531576    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0136143-9e52-4e16-aa8a-0e190e0239e2-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899\" (UID: \"c0136143-9e52-4e16-aa8a-0e190e0239e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899"
Nov 03 00:36:00.531794 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:00.531632    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0136143-9e52-4e16-aa8a-0e190e0239e2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899\" (UID: \"c0136143-9e52-4e16-aa8a-0e190e0239e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899"
Nov 03 00:36:00.531794 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:00.531738    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7686\" (UniqueName: \"kubernetes.io/projected/c0136143-9e52-4e16-aa8a-0e190e0239e2-kube-api-access-c7686\") pod \"maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899\" (UID: \"c0136143-9e52-4e16-aa8a-0e190e0239e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899"
Nov 03 00:36:00.531794 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:36:00.531787    5012 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-serving-cert: secret "maas-api-e2e-shared-b-w2-7ef1b6-serving-cert" not found
Nov 03 00:36:00.531956 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:00.531800    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0136143-9e52-4e16-aa8a-0e190e0239e2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899\" (UID: \"c0136143-9e52-4e16-aa8a-0e190e0239e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899"
Nov 03 00:36:00.531956 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:36:00.531844    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0136143-9e52-4e16-aa8a-0e190e0239e2-maas-api-tls podName:c0136143-9e52-4e16-aa8a-0e190e0239e2 nodeName:}" failed. No retries permitted until 2026-11-03 00:36:01.031830713 +0000 UTC m=+1843.911414749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c0136143-9e52-4e16-aa8a-0e190e0239e2-maas-api-tls") pod "maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899" (UID: "c0136143-9e52-4e16-aa8a-0e190e0239e2") : secret "maas-api-e2e-shared-b-w2-7ef1b6-serving-cert" not found
Nov 03 00:36:00.532451 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:00.532432    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0136143-9e52-4e16-aa8a-0e190e0239e2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899\" (UID: \"c0136143-9e52-4e16-aa8a-0e190e0239e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899"
Nov 03 00:36:00.534062 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:00.534042    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0136143-9e52-4e16-aa8a-0e190e0239e2-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899\" (UID: \"c0136143-9e52-4e16-aa8a-0e190e0239e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899"
Nov 03 00:36:00.551554 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:00.551531    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7686\" (UniqueName: \"kubernetes.io/projected/c0136143-9e52-4e16-aa8a-0e190e0239e2-kube-api-access-c7686\") pod \"maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899\" (UID: \"c0136143-9e52-4e16-aa8a-0e190e0239e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899"
Nov 03 00:36:00.946616 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:00.946575    5012 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 03 00:36:00.946801 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:00.946630    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94" podUID="5795e973-761d-43af-91ea-0451024c0d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 03 00:36:01.041497 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:01.041456    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0136143-9e52-4e16-aa8a-0e190e0239e2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899\" (UID: \"c0136143-9e52-4e16-aa8a-0e190e0239e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899"
Nov 03 00:36:01.043862 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:01.043835    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0136143-9e52-4e16-aa8a-0e190e0239e2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899\" (UID: \"c0136143-9e52-4e16-aa8a-0e190e0239e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899"
Nov 03 00:36:01.342988 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:01.341184    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899"
Nov 03 00:36:01.755491 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:01.754308    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899"]
Nov 03 00:36:01.946576 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:01.946539    5012 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 03 00:36:01.946753 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:01.946594    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94" podUID="5795e973-761d-43af-91ea-0451024c0d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 03 00:36:02.613485 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:02.613447    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899" event={"ID":"c0136143-9e52-4e16-aa8a-0e190e0239e2","Type":"ContainerStarted","Data":"2aa0d72633a420ad684bbbb659bc2edf074c96e91cb1e7094aab345cc340bf63"}
Nov 03 00:36:02.613745 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:02.613724    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899" event={"ID":"c0136143-9e52-4e16-aa8a-0e190e0239e2","Type":"ContainerStarted","Data":"714569585efc7580e46640157aa1c702082cd70126ecee711aaab6ded24aa046"}
Nov 03 00:36:02.615051 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:02.615010    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899"
Nov 03 00:36:02.658966 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:02.658926    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899" podStartSLOduration=2.277912199 podStartE2EDuration="2.658902803s" totalImagesPullingTime="380.990604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:02.655295109 +0000 UTC m=+1845.534879152" watchObservedRunningTime="2026-11-03 00:36:02.658902803 +0000 UTC m=+1845.538486847"
Nov 03 00:36:02.947237 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:02.947151    5012 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 03 00:36:02.947237 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:02.947204    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94" podUID="5795e973-761d-43af-91ea-0451024c0d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 03 00:36:03.124302 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:03.124270    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:36:03.946863 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:03.946831    5012 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 03 00:36:03.947043 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:03.946881    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94" podUID="5795e973-761d-43af-91ea-0451024c0d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 03 00:36:04.569611 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:04.569582    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt"
Nov 03 00:36:04.951638 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:04.951604    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:36:04.952598 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:04.952574    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 00:36:07.364068 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:36:07.364028    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:36:09.639612 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:09.639571    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899"
Nov 03 00:36:18.301323 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:18.301289    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w"]
Nov 03 00:36:18.307169 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:18.307150    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w"
Nov 03 00:36:18.326366 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:18.326339    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w"]
Nov 03 00:36:18.330515 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:18.330491    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-a07a90-serving-cert\""
Nov 03 00:36:18.412301 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:18.412262    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9bafb53-588d-4773-8c0b-aa8682c19d37-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w\" (UID: \"e9bafb53-588d-4773-8c0b-aa8682c19d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w"
Nov 03 00:36:18.412494 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:18.412418    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl8kf\" (UniqueName: \"kubernetes.io/projected/e9bafb53-588d-4773-8c0b-aa8682c19d37-kube-api-access-cl8kf\") pod \"maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w\" (UID: \"e9bafb53-588d-4773-8c0b-aa8682c19d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w"
Nov 03 00:36:18.412574 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:18.412499    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9bafb53-588d-4773-8c0b-aa8682c19d37-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w\" (UID: \"e9bafb53-588d-4773-8c0b-aa8682c19d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w"
Nov 03 00:36:18.412680 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:18.412650    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9bafb53-588d-4773-8c0b-aa8682c19d37-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w\" (UID: \"e9bafb53-588d-4773-8c0b-aa8682c19d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w"
Nov 03 00:36:18.513932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:18.513882    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9bafb53-588d-4773-8c0b-aa8682c19d37-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w\" (UID: \"e9bafb53-588d-4773-8c0b-aa8682c19d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w"
Nov 03 00:36:18.514055 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:18.513974    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl8kf\" (UniqueName: \"kubernetes.io/projected/e9bafb53-588d-4773-8c0b-aa8682c19d37-kube-api-access-cl8kf\") pod \"maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w\" (UID: \"e9bafb53-588d-4773-8c0b-aa8682c19d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w"
Nov 03 00:36:18.514055 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:18.514007    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9bafb53-588d-4773-8c0b-aa8682c19d37-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w\" (UID: \"e9bafb53-588d-4773-8c0b-aa8682c19d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w"
Nov 03 00:36:18.514055 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:18.514044    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9bafb53-588d-4773-8c0b-aa8682c19d37-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w\" (UID: \"e9bafb53-588d-4773-8c0b-aa8682c19d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w"
Nov 03 00:36:18.514598 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:18.514574    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9bafb53-588d-4773-8c0b-aa8682c19d37-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w\" (UID: \"e9bafb53-588d-4773-8c0b-aa8682c19d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w"
Nov 03 00:36:18.516443 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:18.516420    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9bafb53-588d-4773-8c0b-aa8682c19d37-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w\" (UID: \"e9bafb53-588d-4773-8c0b-aa8682c19d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w"
Nov 03 00:36:18.516540 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:18.516474    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9bafb53-588d-4773-8c0b-aa8682c19d37-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w\" (UID: \"e9bafb53-588d-4773-8c0b-aa8682c19d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w"
Nov 03 00:36:18.531186 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:18.531166    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl8kf\" (UniqueName: \"kubernetes.io/projected/e9bafb53-588d-4773-8c0b-aa8682c19d37-kube-api-access-cl8kf\") pod \"maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w\" (UID: \"e9bafb53-588d-4773-8c0b-aa8682c19d37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w"
Nov 03 00:36:18.624143 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:18.623168    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w"
Nov 03 00:36:18.941794 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:36:18.941761    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9bafb53_588d_4773_8c0b_aa8682c19d37.slice/crio-749b86cb0bbc85ca67f1131b589ec6c5942ee39b39d6ff2230a1680970f2a8fa WatchSource:0}: Error finding container 749b86cb0bbc85ca67f1131b589ec6c5942ee39b39d6ff2230a1680970f2a8fa: Status 404 returned error can't find the container with id 749b86cb0bbc85ca67f1131b589ec6c5942ee39b39d6ff2230a1680970f2a8fa
Nov 03 00:36:18.942215 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:18.942191    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w"]
Nov 03 00:36:19.755638 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:19.755596    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w" event={"ID":"e9bafb53-588d-4773-8c0b-aa8682c19d37","Type":"ContainerStarted","Data":"a09ff37d360db30848e163617faf4d0c57755d1b8d082c15530af66f9a0dcfbe"}
Nov 03 00:36:19.756201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:19.755657    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w" event={"ID":"e9bafb53-588d-4773-8c0b-aa8682c19d37","Type":"ContainerStarted","Data":"749b86cb0bbc85ca67f1131b589ec6c5942ee39b39d6ff2230a1680970f2a8fa"}
Nov 03 00:36:19.756201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:19.755686    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w"
Nov 03 00:36:19.794574 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:19.794520    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w" podStartSLOduration=1.398503147 podStartE2EDuration="1.794506023s" totalImagesPullingTime="396.002876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:19.792269025 +0000 UTC m=+1862.671853081" watchObservedRunningTime="2026-11-03 00:36:19.794506023 +0000 UTC m=+1862.674090085"
Nov 03 00:36:19.889326 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:19.889294    5012 scope.go:122] "RemoveContainer" containerID="6057f25729ea95d92f9080a9d1c3924ee893f853e2bcd4d8cf502e76570e12df"
Nov 03 00:36:19.901091 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:19.901061    5012 scope.go:122] "RemoveContainer" containerID="485e256f2acb887dfd2380386c79b28b685c743510d6cae68d58ebe3704b2d2a"
Nov 03 00:36:25.848675 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:25.848639    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w"
Nov 03 00:36:30.210842 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.210805    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"]
Nov 03 00:36:30.220152 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.220130    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.236291 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.236270    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-20109d-openshift-default-dockercfg-ls98l\""
Nov 03 00:36:30.250395 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.250367    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"]
Nov 03 00:36:30.349982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.349948    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-envoy\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.382071 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.350067    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22tdl\" (UniqueName: \"kubernetes.io/projected/cfe5fc6e-4689-49d4-8aa0-861317b1965f-kube-api-access-22tdl\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.382071 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.350138    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-podinfo\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.382071 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.350183    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-workload-certs\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.382071 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.350243    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-credential-socket\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.382071 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.350279    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-token\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.382071 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.350453    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-data\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.382071 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.351167    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istiod-ca-cert\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.382071 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.352586    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-workload-socket\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.479969 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.469870    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-workload-socket\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.479969 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.469979    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-envoy\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.479969 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.470048    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22tdl\" (UniqueName: \"kubernetes.io/projected/cfe5fc6e-4689-49d4-8aa0-861317b1965f-kube-api-access-22tdl\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.479969 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.470117    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-podinfo\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.479969 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.470159    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-workload-certs\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.479969 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.470209    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-credential-socket\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.479969 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.470248    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-token\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.479969 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.470331    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-data\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.479969 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.470443    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istiod-ca-cert\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.479969 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.471397    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istiod-ca-cert\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.479969 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.471706    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-workload-socket\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.479969 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.472365    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-workload-certs\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.479969 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.473590    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-credential-socket\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.479969 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.473873    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-data\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.503467 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.493234    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-envoy\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.515409 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.514131    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-token\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.544693 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.533878    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-podinfo\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.544693 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.544624    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22tdl\" (UniqueName: \"kubernetes.io/projected/cfe5fc6e-4689-49d4-8aa0-861317b1965f-kube-api-access-22tdl\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.840827 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.840752    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:30.982399 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.982363    5012 generic.go:356] "Generic (PLEG): container finished" podID="dbbcfc46-aa01-443c-a475-a757649f86ce" containerID="bc2f9bac5940460d145b142b868b1dcd75e0d4c5bdd5f1542e0d841235297be2" exitCode=0
Nov 03 00:36:30.982616 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.982438    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd" event={"ID":"dbbcfc46-aa01-443c-a475-a757649f86ce","Type":"ContainerDied","Data":"bc2f9bac5940460d145b142b868b1dcd75e0d4c5bdd5f1542e0d841235297be2"}
Nov 03 00:36:30.982877 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.982858    5012 scope.go:122] "RemoveContainer" containerID="bc2f9bac5940460d145b142b868b1dcd75e0d4c5bdd5f1542e0d841235297be2"
Nov 03 00:36:30.984525 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.984493    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:36:30.984652 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.984564    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:36:30.984652 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:30.984621    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:36:31.160977 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:36:31.160893    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe5fc6e_4689_49d4_8aa0_861317b1965f.slice/crio-80aad6f3e207312b5140cbe64c0ac46cd6321729f229a319b470a00ddfc3ae00 WatchSource:0}: Error finding container 80aad6f3e207312b5140cbe64c0ac46cd6321729f229a319b470a00ddfc3ae00: Status 404 returned error can't find the container with id 80aad6f3e207312b5140cbe64c0ac46cd6321729f229a319b470a00ddfc3ae00
Nov 03 00:36:31.161534 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:31.161506    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"]
Nov 03 00:36:31.170126 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:31.169709    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:36:31.170126 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:31.169804    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:36:31.171528 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:31.170396    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:36:32.015849 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:32.015809    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd" event={"ID":"cfe5fc6e-4689-49d4-8aa0-861317b1965f","Type":"ContainerStarted","Data":"11a3b0c7e452e6df6e38322dbba7f28232631ff37acf03cf42122850dccf77b8"}
Nov 03 00:36:32.016389 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:32.015868    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd" event={"ID":"cfe5fc6e-4689-49d4-8aa0-861317b1965f","Type":"ContainerStarted","Data":"80aad6f3e207312b5140cbe64c0ac46cd6321729f229a319b470a00ddfc3ae00"}
Nov 03 00:36:32.018170 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:32.018144    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd" event={"ID":"dbbcfc46-aa01-443c-a475-a757649f86ce","Type":"ContainerStarted","Data":"c83cb7eaf2e28ef1ab4cf8455339c11b76f959fa66a8f50b1d84f303f7c07188"}
Nov 03 00:36:32.067709 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:32.067661    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd" podStartSLOduration=2.06764712 podStartE2EDuration="2.06764712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:32.064623975 +0000 UTC m=+1874.944208019" watchObservedRunningTime="2026-11-03 00:36:32.06764712 +0000 UTC m=+1874.947231163"
Nov 03 00:36:32.706934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:32.706228    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:36:32.710934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:32.710009    5012 patch_prober.go:29] interesting pod/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Nov 03 00:36:32.710934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:32.710065    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd" podUID="dbbcfc46-aa01-443c-a475-a757649f86ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Nov 03 00:36:32.841072 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:32.841040    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:32.846097 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:32.846071    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:33.057986 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:33.055699    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:33.065323 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:33.063177    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:36:33.707312 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:33.707272    5012 patch_prober.go:29] interesting pod/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Nov 03 00:36:33.707608 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:33.707578    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd" podUID="dbbcfc46-aa01-443c-a475-a757649f86ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Nov 03 00:36:34.712099 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:34.712054    5012 patch_prober.go:29] interesting pod/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Nov 03 00:36:34.712536 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:34.712148    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd" podUID="dbbcfc46-aa01-443c-a475-a757649f86ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Nov 03 00:36:35.708933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:35.708237    5012 patch_prober.go:29] interesting pod/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Nov 03 00:36:35.708933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:35.708311    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd" podUID="dbbcfc46-aa01-443c-a475-a757649f86ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Nov 03 00:36:36.156867 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:36.156832    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:36:36.710248 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:36.710208    5012 patch_prober.go:29] interesting pod/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Nov 03 00:36:36.710551 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:36.710512    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd" podUID="dbbcfc46-aa01-443c-a475-a757649f86ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Nov 03 00:36:37.380044 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:36:37.380007    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:36:37.898049 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:37.898008    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:36:37.928934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:37.920756    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 00:36:44.439770 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.439733    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"]
Nov 03 00:36:44.444404 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.444385    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:36:44.460407 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.460380    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-065de0\"/\"kube-root-ca.crt\""
Nov 03 00:36:44.470957 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.470897    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-065de0\"/\"auth-test-model-065de0-kserve-self-signed-certs\""
Nov 03 00:36:44.471113 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.470992    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-065de0\"/\"default-dockercfg-897q9\""
Nov 03 00:36:44.471298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.471284    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-065de0\"/\"openshift-service-ca.crt\""
Nov 03 00:36:44.495146 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.495120    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"]
Nov 03 00:36:44.499423 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.499395    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-home\") pod \"auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g\" (UID: \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\") " pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:36:44.499567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.499447    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sdxv\" (UniqueName: \"kubernetes.io/projected/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-kube-api-access-6sdxv\") pod \"auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g\" (UID: \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\") " pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:36:44.499567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.499479    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-tls-certs\") pod \"auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g\" (UID: \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\") " pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:36:44.499688 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.499603    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-model-cache\") pod \"auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g\" (UID: \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\") " pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:36:44.499773 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.499752    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-tmp-dir\") pod \"auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g\" (UID: \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\") " pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:36:44.499826 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.499813    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-dshm\") pod \"auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g\" (UID: \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\") " pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:36:44.601497 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.601452    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-home\") pod \"auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g\" (UID: \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\") " pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:36:44.601761 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.601507    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sdxv\" (UniqueName: \"kubernetes.io/projected/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-kube-api-access-6sdxv\") pod \"auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g\" (UID: \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\") " pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:36:44.601761 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.601677    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-tls-certs\") pod \"auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g\" (UID: \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\") " pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:36:44.601761 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.601735    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-model-cache\") pod \"auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g\" (UID: \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\") " pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:36:44.601970 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.601904    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-tmp-dir\") pod \"auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g\" (UID: \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\") " pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:36:44.602036 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.602009    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-dshm\") pod \"auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g\" (UID: \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\") " pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:36:44.602099 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.601905    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-home\") pod \"auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g\" (UID: \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\") " pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:36:44.602212 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.602190    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-model-cache\") pod \"auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g\" (UID: \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\") " pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:36:44.602336 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.602213    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-tmp-dir\") pod \"auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g\" (UID: \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\") " pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:36:44.604114 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.604091    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-dshm\") pod \"auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g\" (UID: \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\") " pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:36:44.604441 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.604418    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-tls-certs\") pod \"auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g\" (UID: \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\") " pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:36:44.620675 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.620651    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sdxv\" (UniqueName: \"kubernetes.io/projected/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-kube-api-access-6sdxv\") pod \"auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g\" (UID: \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\") " pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:36:44.771727 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:44.770970    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:36:45.199390 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:36:45.199360    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc53fb3c_4315_4804_b73e_c4c44c9fbca7.slice/crio-b95a823e8fb391d368f2b5dadec93fba6cc91d882ca85f437b32e0ce1d04484c WatchSource:0}: Error finding container b95a823e8fb391d368f2b5dadec93fba6cc91d882ca85f437b32e0ce1d04484c: Status 404 returned error can't find the container with id b95a823e8fb391d368f2b5dadec93fba6cc91d882ca85f437b32e0ce1d04484c
Nov 03 00:36:45.199984 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:45.199867    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"]
Nov 03 00:36:45.247489 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:45.247453    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g" event={"ID":"bc53fb3c-4315-4804-b73e-c4c44c9fbca7","Type":"ContainerStarted","Data":"b95a823e8fb391d368f2b5dadec93fba6cc91d882ca85f437b32e0ce1d04484c"}
Nov 03 00:36:46.308205 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:46.306934    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g" event={"ID":"bc53fb3c-4315-4804-b73e-c4c44c9fbca7","Type":"ContainerStarted","Data":"818fd94c77d6023e3127ed72fcc9fad31a132ff443bcdc7c949c2fb10f5508d7"}
Nov 03 00:36:46.370065 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:46.368856    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g" podStartSLOduration=2.368835994 podStartE2EDuration="2.368835994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:46.353522309 +0000 UTC m=+1889.233106353" watchObservedRunningTime="2026-11-03 00:36:46.368835994 +0000 UTC m=+1889.248420047"
Nov 03 00:36:46.772553 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:46.772514    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:36:48.327334 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:48.327297    5012 generic.go:356] "Generic (PLEG): container finished" podID="209af59a-bfd0-4aab-b2bf-7c789190f57e" containerID="5c0755ea0d94d72ad025aa98c515eb99c1a341bd0a7db744155a383853f2ecf8" exitCode=0
Nov 03 00:36:48.327880 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:48.327350    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k" event={"ID":"209af59a-bfd0-4aab-b2bf-7c789190f57e","Type":"ContainerDied","Data":"5c0755ea0d94d72ad025aa98c515eb99c1a341bd0a7db744155a383853f2ecf8"}
Nov 03 00:36:48.327880 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:48.327820    5012 scope.go:122] "RemoveContainer" containerID="5c0755ea0d94d72ad025aa98c515eb99c1a341bd0a7db744155a383853f2ecf8"
Nov 03 00:36:48.329582 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:48.329548    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:36:48.329673 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:48.329642    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:36:48.329721 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:48.329692    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:36:49.339250 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:49.339209    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k" event={"ID":"209af59a-bfd0-4aab-b2bf-7c789190f57e","Type":"ContainerStarted","Data":"8c03ea4f5fcb71e7715b3847754388dd36f36878eb7fa0a104e70d6e9a7b73c2"}
Nov 03 00:36:49.466141 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:49.466104    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:36:49.466360 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:49.466305    5012 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 03 00:36:49.466455 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:49.466355    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k" podUID="209af59a-bfd0-4aab-b2bf-7c789190f57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 03 00:36:50.465459 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:50.465424    5012 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 03 00:36:50.465971 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:50.465475    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k" podUID="209af59a-bfd0-4aab-b2bf-7c789190f57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 03 00:36:51.465716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:51.465676    5012 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 03 00:36:51.466146 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:51.465744    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k" podUID="209af59a-bfd0-4aab-b2bf-7c789190f57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 03 00:36:52.466035 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:52.465997    5012 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 03 00:36:52.466470 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:52.466061    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k" podUID="209af59a-bfd0-4aab-b2bf-7c789190f57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 03 00:36:53.465310 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:53.465274    5012 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 03 00:36:53.465521 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:53.465324    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k" podUID="209af59a-bfd0-4aab-b2bf-7c789190f57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 03 00:36:54.466038 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:54.466002    5012 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 03 00:36:54.466435 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:54.466056    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k" podUID="209af59a-bfd0-4aab-b2bf-7c789190f57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 03 00:36:54.480139 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:54.480105    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:36:54.773121 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:54.773042    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:36:54.787860 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:54.787829    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:36:55.316574 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.316531    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"]
Nov 03 00:36:55.323300 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.323275    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:36:55.340847 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.340821    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-tmp-dir\") pod \"auth-test-model-359f71-kserve-859cfd64d5-p87rq\" (UID: \"3bff9372-676b-47d2-89e4-f423a422cf1d\") " pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:36:55.341047 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.341016    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bff9372-676b-47d2-89e4-f423a422cf1d-tls-certs\") pod \"auth-test-model-359f71-kserve-859cfd64d5-p87rq\" (UID: \"3bff9372-676b-47d2-89e4-f423a422cf1d\") " pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:36:55.341331 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.341308    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzft9\" (UniqueName: \"kubernetes.io/projected/3bff9372-676b-47d2-89e4-f423a422cf1d-kube-api-access-dzft9\") pod \"auth-test-model-359f71-kserve-859cfd64d5-p87rq\" (UID: \"3bff9372-676b-47d2-89e4-f423a422cf1d\") " pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:36:55.341422 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.341364    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-model-cache\") pod \"auth-test-model-359f71-kserve-859cfd64d5-p87rq\" (UID: \"3bff9372-676b-47d2-89e4-f423a422cf1d\") " pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:36:55.341510 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.341487    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-home\") pod \"auth-test-model-359f71-kserve-859cfd64d5-p87rq\" (UID: \"3bff9372-676b-47d2-89e4-f423a422cf1d\") " pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:36:55.341681 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.341647    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-dshm\") pod \"auth-test-model-359f71-kserve-859cfd64d5-p87rq\" (UID: \"3bff9372-676b-47d2-89e4-f423a422cf1d\") " pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:36:55.345594 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.341963    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"]
Nov 03 00:36:55.345594 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.344231    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-359f71\"/\"default-dockercfg-w25h2\""
Nov 03 00:36:55.345594 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.344418    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-359f71\"/\"auth-test-model-359f71-kserve-self-signed-certs\""
Nov 03 00:36:55.345594 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.344576    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-359f71\"/\"openshift-service-ca.crt\""
Nov 03 00:36:55.350507 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.350483    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-359f71\"/\"kube-root-ca.crt\""
Nov 03 00:36:55.399203 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.399178    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:36:55.442759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.442721    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bff9372-676b-47d2-89e4-f423a422cf1d-tls-certs\") pod \"auth-test-model-359f71-kserve-859cfd64d5-p87rq\" (UID: \"3bff9372-676b-47d2-89e4-f423a422cf1d\") " pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:36:55.442948 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.442853    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzft9\" (UniqueName: \"kubernetes.io/projected/3bff9372-676b-47d2-89e4-f423a422cf1d-kube-api-access-dzft9\") pod \"auth-test-model-359f71-kserve-859cfd64d5-p87rq\" (UID: \"3bff9372-676b-47d2-89e4-f423a422cf1d\") " pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:36:55.442948 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.442920    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-model-cache\") pod \"auth-test-model-359f71-kserve-859cfd64d5-p87rq\" (UID: \"3bff9372-676b-47d2-89e4-f423a422cf1d\") " pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:36:55.443092 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.442987    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-home\") pod \"auth-test-model-359f71-kserve-859cfd64d5-p87rq\" (UID: \"3bff9372-676b-47d2-89e4-f423a422cf1d\") " pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:36:55.443092 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.443075    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-dshm\") pod \"auth-test-model-359f71-kserve-859cfd64d5-p87rq\" (UID: \"3bff9372-676b-47d2-89e4-f423a422cf1d\") " pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:36:55.443261 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.443238    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-tmp-dir\") pod \"auth-test-model-359f71-kserve-859cfd64d5-p87rq\" (UID: \"3bff9372-676b-47d2-89e4-f423a422cf1d\") " pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:36:55.443557 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.443534    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-home\") pod \"auth-test-model-359f71-kserve-859cfd64d5-p87rq\" (UID: \"3bff9372-676b-47d2-89e4-f423a422cf1d\") " pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:36:55.443859 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.443803    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-model-cache\") pod \"auth-test-model-359f71-kserve-859cfd64d5-p87rq\" (UID: \"3bff9372-676b-47d2-89e4-f423a422cf1d\") " pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:36:55.443986 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.443874    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-tmp-dir\") pod \"auth-test-model-359f71-kserve-859cfd64d5-p87rq\" (UID: \"3bff9372-676b-47d2-89e4-f423a422cf1d\") " pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:36:55.445415 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.445395    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-dshm\") pod \"auth-test-model-359f71-kserve-859cfd64d5-p87rq\" (UID: \"3bff9372-676b-47d2-89e4-f423a422cf1d\") " pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:36:55.445674 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.445653    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bff9372-676b-47d2-89e4-f423a422cf1d-tls-certs\") pod \"auth-test-model-359f71-kserve-859cfd64d5-p87rq\" (UID: \"3bff9372-676b-47d2-89e4-f423a422cf1d\") " pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:36:55.461831 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.461804    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzft9\" (UniqueName: \"kubernetes.io/projected/3bff9372-676b-47d2-89e4-f423a422cf1d-kube-api-access-dzft9\") pod \"auth-test-model-359f71-kserve-859cfd64d5-p87rq\" (UID: \"3bff9372-676b-47d2-89e4-f423a422cf1d\") " pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:36:55.471738 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.471714    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:36:55.473133 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.473115    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 00:36:55.641013 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:55.640983    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:36:56.090663 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:56.090635    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"]
Nov 03 00:36:56.090810 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:36:56.090727    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bff9372_676b_47d2_89e4_f423a422cf1d.slice/crio-7b6bdedb4476b2a7c6f133554cc6afecde635a0c3f9ea268da32befa50bc537b WatchSource:0}: Error finding container 7b6bdedb4476b2a7c6f133554cc6afecde635a0c3f9ea268da32befa50bc537b: Status 404 returned error can't find the container with id 7b6bdedb4476b2a7c6f133554cc6afecde635a0c3f9ea268da32befa50bc537b
Nov 03 00:36:56.399365 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:56.399314    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq" event={"ID":"3bff9372-676b-47d2-89e4-f423a422cf1d","Type":"ContainerStarted","Data":"11076b1a0ed2174e8f11a2bda8e8f8dfb13b7133cd9cf06d33e20a624b369d69"}
Nov 03 00:36:56.399365 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:56.399369    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq" event={"ID":"3bff9372-676b-47d2-89e4-f423a422cf1d","Type":"ContainerStarted","Data":"7b6bdedb4476b2a7c6f133554cc6afecde635a0c3f9ea268da32befa50bc537b"}
Nov 03 00:36:56.437809 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:56.437748    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq" podStartSLOduration=1.437724218 podStartE2EDuration="1.437724218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:56.435948993 +0000 UTC m=+1899.315533036" watchObservedRunningTime="2026-11-03 00:36:56.437724218 +0000 UTC m=+1899.317308262"
Nov 03 00:36:56.642069 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:36:56.642037    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:37:05.641609 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:05.641566    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:37:05.655380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:05.655348    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:37:06.708228 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:06.708196    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:37:07.366547 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:37:07.366504    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:37:35.207989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:35.207264    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4"]
Nov 03 00:37:37.365558 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:37:37.365522    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:37:40.208042 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:40.207957    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4" podUID="013f254b-241b-41e8-9974-56b75b830262" containerName="payload-processing" containerID="cri-o://6da04edb1ddcbf5687edc859e8fe5343a43a387aecde41dd47ff8505d518b083" gracePeriod=25
Nov 03 00:37:40.536085 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:40.536057    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4"
Nov 03 00:37:40.674855 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:40.674828    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/013f254b-241b-41e8-9974-56b75b830262-kube-api-access-wgwvj\" (UniqueName: \"kubernetes.io/projected/013f254b-241b-41e8-9974-56b75b830262-kube-api-access-wgwvj\") pod \"013f254b-241b-41e8-9974-56b75b830262\" (UID: \"013f254b-241b-41e8-9974-56b75b830262\") "
Nov 03 00:37:40.675035 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:40.675013    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/013f254b-241b-41e8-9974-56b75b830262-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/013f254b-241b-41e8-9974-56b75b830262-plugins-config-volume\") pod \"013f254b-241b-41e8-9974-56b75b830262\" (UID: \"013f254b-241b-41e8-9974-56b75b830262\") "
Nov 03 00:37:40.675401 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:40.675370    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/013f254b-241b-41e8-9974-56b75b830262-plugins-config-volume" pod "013f254b-241b-41e8-9974-56b75b830262" (UID: "013f254b-241b-41e8-9974-56b75b830262"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:37:40.675633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:40.675616    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/013f254b-241b-41e8-9974-56b75b830262-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:37:40.676905 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:40.676881    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013f254b-241b-41e8-9974-56b75b830262-kube-api-access-wgwvj" pod "013f254b-241b-41e8-9974-56b75b830262" (UID: "013f254b-241b-41e8-9974-56b75b830262"). InnerVolumeSpecName "kube-api-access-wgwvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:37:40.776683 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:40.776623    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgwvj\" (UniqueName: \"kubernetes.io/projected/013f254b-241b-41e8-9974-56b75b830262-kube-api-access-wgwvj\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:37:40.971099 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:40.971060    5012 generic.go:356] "Generic (PLEG): container finished" podID="013f254b-241b-41e8-9974-56b75b830262" containerID="6da04edb1ddcbf5687edc859e8fe5343a43a387aecde41dd47ff8505d518b083" exitCode=0
Nov 03 00:37:40.971099 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:40.971105    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4" event={"ID":"013f254b-241b-41e8-9974-56b75b830262","Type":"ContainerDied","Data":"6da04edb1ddcbf5687edc859e8fe5343a43a387aecde41dd47ff8505d518b083"}
Nov 03 00:37:40.971357 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:40.971112    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4"
Nov 03 00:37:40.971357 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:40.971155    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4" event={"ID":"013f254b-241b-41e8-9974-56b75b830262","Type":"ContainerDied","Data":"e4287cc109e3338138196361993655604b8be08b8e36296d3b14c47321f6e7be"}
Nov 03 00:37:40.971357 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:40.971182    5012 scope.go:122] "RemoveContainer" containerID="6da04edb1ddcbf5687edc859e8fe5343a43a387aecde41dd47ff8505d518b083"
Nov 03 00:37:40.986049 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:40.986026    5012 scope.go:122] "RemoveContainer" containerID="6da04edb1ddcbf5687edc859e8fe5343a43a387aecde41dd47ff8505d518b083"
Nov 03 00:37:40.986354 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:37:40.986323    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da04edb1ddcbf5687edc859e8fe5343a43a387aecde41dd47ff8505d518b083\": container with ID starting with 6da04edb1ddcbf5687edc859e8fe5343a43a387aecde41dd47ff8505d518b083 not found: ID does not exist" containerID="6da04edb1ddcbf5687edc859e8fe5343a43a387aecde41dd47ff8505d518b083"
Nov 03 00:37:40.986419 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:40.986364    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da04edb1ddcbf5687edc859e8fe5343a43a387aecde41dd47ff8505d518b083"} err="failed to get container status \"6da04edb1ddcbf5687edc859e8fe5343a43a387aecde41dd47ff8505d518b083\": rpc error: code = NotFound desc = could not find container \"6da04edb1ddcbf5687edc859e8fe5343a43a387aecde41dd47ff8505d518b083\": container with ID starting with 6da04edb1ddcbf5687edc859e8fe5343a43a387aecde41dd47ff8505d518b083 not found: ID does not exist"
Nov 03 00:37:41.023387 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:41.023359    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4"]
Nov 03 00:37:41.033508 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:41.033450    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c59124-54f6b5ffb9-6gxh4"]
Nov 03 00:37:41.249380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:41.249343    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="013f254b-241b-41e8-9974-56b75b830262" path="/var/lib/kubelet/pods/013f254b-241b-41e8-9974-56b75b830262/volumes"
Nov 03 00:37:42.270314 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.270275    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"]
Nov 03 00:37:42.271288 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.271265    5012 cpu_manager.go:414] "containerMap: removing container" podUID="013f254b-241b-41e8-9974-56b75b830262" containerName="payload-processing"
Nov 03 00:37:42.271357 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.271291    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="013f254b-241b-41e8-9974-56b75b830262" containerName="payload-processing"
Nov 03 00:37:42.271559 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.271544    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="013f254b-241b-41e8-9974-56b75b830262" containerName="payload-processing"
Nov 03 00:37:42.278475 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.278444    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:37:42.292755 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.292713    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-2858fb\"/\"kube-root-ca.crt\""
Nov 03 00:37:42.295858 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.295828    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"]
Nov 03 00:37:42.299516 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.299495    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-2858fb\"/\"auth-test-model-2858fb-kserve-self-signed-certs\""
Nov 03 00:37:42.299722 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.299698    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-2858fb\"/\"openshift-service-ca.crt\""
Nov 03 00:37:42.299833 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.299795    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-2858fb\"/\"default-dockercfg-7tbcg\""
Nov 03 00:37:42.311625 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.311591    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-tls-certs\") pod \"auth-test-model-2858fb-kserve-7946c49589-fjnfl\" (UID: \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\") " pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:37:42.311754 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.311643    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rvkl\" (UniqueName: \"kubernetes.io/projected/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-kube-api-access-4rvkl\") pod \"auth-test-model-2858fb-kserve-7946c49589-fjnfl\" (UID: \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\") " pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:37:42.311827 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.311776    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-tmp-dir\") pod \"auth-test-model-2858fb-kserve-7946c49589-fjnfl\" (UID: \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\") " pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:37:42.311925 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.311886    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-model-cache\") pod \"auth-test-model-2858fb-kserve-7946c49589-fjnfl\" (UID: \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\") " pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:37:42.312095 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.312073    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-dshm\") pod \"auth-test-model-2858fb-kserve-7946c49589-fjnfl\" (UID: \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\") " pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:37:42.312233 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.312158    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-home\") pod \"auth-test-model-2858fb-kserve-7946c49589-fjnfl\" (UID: \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\") " pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:37:42.413962 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.413919    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-tmp-dir\") pod \"auth-test-model-2858fb-kserve-7946c49589-fjnfl\" (UID: \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\") " pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:37:42.414121 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.413993    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-model-cache\") pod \"auth-test-model-2858fb-kserve-7946c49589-fjnfl\" (UID: \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\") " pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:37:42.414121 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.414073    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-dshm\") pod \"auth-test-model-2858fb-kserve-7946c49589-fjnfl\" (UID: \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\") " pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:37:42.414121 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.414119    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-home\") pod \"auth-test-model-2858fb-kserve-7946c49589-fjnfl\" (UID: \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\") " pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:37:42.414323 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.414185    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-tls-certs\") pod \"auth-test-model-2858fb-kserve-7946c49589-fjnfl\" (UID: \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\") " pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:37:42.414323 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.414301    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rvkl\" (UniqueName: \"kubernetes.io/projected/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-kube-api-access-4rvkl\") pod \"auth-test-model-2858fb-kserve-7946c49589-fjnfl\" (UID: \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\") " pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:37:42.414450 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.414308    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-tmp-dir\") pod \"auth-test-model-2858fb-kserve-7946c49589-fjnfl\" (UID: \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\") " pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:37:42.414535 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.414512    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-home\") pod \"auth-test-model-2858fb-kserve-7946c49589-fjnfl\" (UID: \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\") " pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:37:42.414602 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.414570    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-model-cache\") pod \"auth-test-model-2858fb-kserve-7946c49589-fjnfl\" (UID: \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\") " pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:37:42.416318 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.416292    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-dshm\") pod \"auth-test-model-2858fb-kserve-7946c49589-fjnfl\" (UID: \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\") " pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:37:42.416569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.416550    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-tls-certs\") pod \"auth-test-model-2858fb-kserve-7946c49589-fjnfl\" (UID: \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\") " pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:37:42.431272 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.431250    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rvkl\" (UniqueName: \"kubernetes.io/projected/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-kube-api-access-4rvkl\") pod \"auth-test-model-2858fb-kserve-7946c49589-fjnfl\" (UID: \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\") " pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:37:42.612295 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.609723    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:37:42.941043 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:37:42.941015    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a1788b2_60c1_4e37_845b_879b4b5ef3f7.slice/crio-2a3d252ddacb90da329dd831e27de6c7735893c1b82515940340cdab77eb8726 WatchSource:0}: Error finding container 2a3d252ddacb90da329dd831e27de6c7735893c1b82515940340cdab77eb8726: Status 404 returned error can't find the container with id 2a3d252ddacb90da329dd831e27de6c7735893c1b82515940340cdab77eb8726
Nov 03 00:37:42.941043 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:42.941029    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"]
Nov 03 00:37:43.022885 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:43.022850    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl" event={"ID":"2a1788b2-60c1-4e37-845b-879b4b5ef3f7","Type":"ContainerStarted","Data":"2a3d252ddacb90da329dd831e27de6c7735893c1b82515940340cdab77eb8726"}
Nov 03 00:37:44.032325 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:44.032233    5012 generic.go:356] "Generic (PLEG): container finished" podID="cfe5fc6e-4689-49d4-8aa0-861317b1965f" containerID="11a3b0c7e452e6df6e38322dbba7f28232631ff37acf03cf42122850dccf77b8" exitCode=0
Nov 03 00:37:44.032799 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:44.032312    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd" event={"ID":"cfe5fc6e-4689-49d4-8aa0-861317b1965f","Type":"ContainerDied","Data":"11a3b0c7e452e6df6e38322dbba7f28232631ff37acf03cf42122850dccf77b8"}
Nov 03 00:37:44.032799 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:44.032760    5012 scope.go:122] "RemoveContainer" containerID="11a3b0c7e452e6df6e38322dbba7f28232631ff37acf03cf42122850dccf77b8"
Nov 03 00:37:44.034080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:44.034054    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl" event={"ID":"2a1788b2-60c1-4e37-845b-879b4b5ef3f7","Type":"ContainerStarted","Data":"443fa5965d757eac55a8efde91b1c00959aec45711ebb0350e568d86fbf79e86"}
Nov 03 00:37:44.034653 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:44.034622    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:37:44.034732 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:44.034695    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:37:44.034779 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:44.034732    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:37:44.128982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:44.128830    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl" podStartSLOduration=2.128812231 podStartE2EDuration="2.128812231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:37:44.124209514 +0000 UTC m=+1947.003793560" watchObservedRunningTime="2026-11-03 00:37:44.128812231 +0000 UTC m=+1947.008396277"
Nov 03 00:37:44.610621 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:44.610522    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:37:45.044892 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:45.044801    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd" event={"ID":"cfe5fc6e-4689-49d4-8aa0-861317b1965f","Type":"ContainerStarted","Data":"c21a21bb579679b595176c35b4748a85b9058f734621ae3dd50c5d719bb30778"}
Nov 03 00:37:45.841121 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:45.841086    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:37:45.842388 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:45.842360    5012 patch_prober.go:29] interesting pod/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Nov 03 00:37:45.842523 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:45.842423    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd" podUID="cfe5fc6e-4689-49d4-8aa0-861317b1965f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Nov 03 00:37:46.841096 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:46.841061    5012 patch_prober.go:29] interesting pod/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Nov 03 00:37:46.841613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:46.841131    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd" podUID="cfe5fc6e-4689-49d4-8aa0-861317b1965f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Nov 03 00:37:47.841063 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:47.841026    5012 patch_prober.go:29] interesting pod/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Nov 03 00:37:47.841269 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:47.841087    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd" podUID="cfe5fc6e-4689-49d4-8aa0-861317b1965f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Nov 03 00:37:48.055405 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:48.055360    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:37:48.844379 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:48.844342    5012 patch_prober.go:29] interesting pod/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Nov 03 00:37:48.845029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:48.844993    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd" podUID="cfe5fc6e-4689-49d4-8aa0-861317b1965f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Nov 03 00:37:49.841365 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:49.841216    5012 patch_prober.go:29] interesting pod/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Nov 03 00:37:49.841365 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:49.841279    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd" podUID="cfe5fc6e-4689-49d4-8aa0-861317b1965f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Nov 03 00:37:50.846404 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:50.846373    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:37:50.847533 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:50.847508    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 00:37:51.291496 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:51.291464    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6"]
Nov 03 00:37:51.291900 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:51.291864    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6" podUID="589bae50-3c0e-466e-849b-cd6052804d65" containerName="maas-api" containerID="cri-o://fc853407f2a11c3ce5ea4a56868afd0448b2fff90c9b8c537157349dfb1165bf" gracePeriod=30
Nov 03 00:37:51.772611 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:51.772087    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6"
Nov 03 00:37:51.919141 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:51.919057    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/589bae50-3c0e-466e-849b-cd6052804d65-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/589bae50-3c0e-466e-849b-cd6052804d65-maas-api-tls\") pod \"589bae50-3c0e-466e-849b-cd6052804d65\" (UID: \"589bae50-3c0e-466e-849b-cd6052804d65\") "
Nov 03 00:37:51.919141 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:51.919113    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/589bae50-3c0e-466e-849b-cd6052804d65-metrics-certs\" (UniqueName: \"kubernetes.io/secret/589bae50-3c0e-466e-849b-cd6052804d65-metrics-certs\") pod \"589bae50-3c0e-466e-849b-cd6052804d65\" (UID: \"589bae50-3c0e-466e-849b-cd6052804d65\") "
Nov 03 00:37:51.919633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:51.919207    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/589bae50-3c0e-466e-849b-cd6052804d65-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/589bae50-3c0e-466e-849b-cd6052804d65-openshift-service-ca\") pod \"589bae50-3c0e-466e-849b-cd6052804d65\" (UID: \"589bae50-3c0e-466e-849b-cd6052804d65\") "
Nov 03 00:37:51.919633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:51.919395    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/589bae50-3c0e-466e-849b-cd6052804d65-kube-api-access-w5bnr\" (UniqueName: \"kubernetes.io/projected/589bae50-3c0e-466e-849b-cd6052804d65-kube-api-access-w5bnr\") pod \"589bae50-3c0e-466e-849b-cd6052804d65\" (UID: \"589bae50-3c0e-466e-849b-cd6052804d65\") "
Nov 03 00:37:51.919633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:51.919581    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/589bae50-3c0e-466e-849b-cd6052804d65-openshift-service-ca" pod "589bae50-3c0e-466e-849b-cd6052804d65" (UID: "589bae50-3c0e-466e-849b-cd6052804d65"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:37:51.920389 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:51.920351    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/589bae50-3c0e-466e-849b-cd6052804d65-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:37:51.921513 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:51.921487    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/589bae50-3c0e-466e-849b-cd6052804d65-maas-api-tls" pod "589bae50-3c0e-466e-849b-cd6052804d65" (UID: "589bae50-3c0e-466e-849b-cd6052804d65"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:37:51.921997 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:51.921969    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/589bae50-3c0e-466e-849b-cd6052804d65-kube-api-access-w5bnr" pod "589bae50-3c0e-466e-849b-cd6052804d65" (UID: "589bae50-3c0e-466e-849b-cd6052804d65"). InnerVolumeSpecName "kube-api-access-w5bnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:37:51.922610 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:51.922583    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/589bae50-3c0e-466e-849b-cd6052804d65-metrics-certs" pod "589bae50-3c0e-466e-849b-cd6052804d65" (UID: "589bae50-3c0e-466e-849b-cd6052804d65"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:37:52.021631 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:52.021602    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/589bae50-3c0e-466e-849b-cd6052804d65-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:37:52.021631 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:52.021629    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/589bae50-3c0e-466e-849b-cd6052804d65-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:37:52.021631 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:52.021639    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5bnr\" (UniqueName: \"kubernetes.io/projected/589bae50-3c0e-466e-849b-cd6052804d65-kube-api-access-w5bnr\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:37:52.103759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:52.103726    5012 generic.go:356] "Generic (PLEG): container finished" podID="589bae50-3c0e-466e-849b-cd6052804d65" containerID="fc853407f2a11c3ce5ea4a56868afd0448b2fff90c9b8c537157349dfb1165bf" exitCode=0
Nov 03 00:37:52.103902 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:52.103782    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6"
Nov 03 00:37:52.103902 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:52.103811    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6" event={"ID":"589bae50-3c0e-466e-849b-cd6052804d65","Type":"ContainerDied","Data":"fc853407f2a11c3ce5ea4a56868afd0448b2fff90c9b8c537157349dfb1165bf"}
Nov 03 00:37:52.103902 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:52.103870    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6" event={"ID":"589bae50-3c0e-466e-849b-cd6052804d65","Type":"ContainerDied","Data":"67adb3eefcba6c93ac6b6e8c0a157019c8dd839d1fa523b669da724b7708c193"}
Nov 03 00:37:52.103902 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:52.103893    5012 scope.go:122] "RemoveContainer" containerID="fc853407f2a11c3ce5ea4a56868afd0448b2fff90c9b8c537157349dfb1165bf"
Nov 03 00:37:52.115522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:52.115503    5012 scope.go:122] "RemoveContainer" containerID="fc853407f2a11c3ce5ea4a56868afd0448b2fff90c9b8c537157349dfb1165bf"
Nov 03 00:37:52.115793 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:37:52.115775    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc853407f2a11c3ce5ea4a56868afd0448b2fff90c9b8c537157349dfb1165bf\": container with ID starting with fc853407f2a11c3ce5ea4a56868afd0448b2fff90c9b8c537157349dfb1165bf not found: ID does not exist" containerID="fc853407f2a11c3ce5ea4a56868afd0448b2fff90c9b8c537157349dfb1165bf"
Nov 03 00:37:52.115834 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:52.115800    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc853407f2a11c3ce5ea4a56868afd0448b2fff90c9b8c537157349dfb1165bf"} err="failed to get container status \"fc853407f2a11c3ce5ea4a56868afd0448b2fff90c9b8c537157349dfb1165bf\": rpc error: code = NotFound desc = could not find container \"fc853407f2a11c3ce5ea4a56868afd0448b2fff90c9b8c537157349dfb1165bf\": container with ID starting with fc853407f2a11c3ce5ea4a56868afd0448b2fff90c9b8c537157349dfb1165bf not found: ID does not exist"
Nov 03 00:37:52.154449 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:52.154425    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6"]
Nov 03 00:37:52.165045 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:52.165023    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c59124-7fd9b6dc44-wxmm6"]
Nov 03 00:37:52.611250 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:52.611216    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:37:52.624193 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:52.624168    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:37:53.124668 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:53.124635    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:37:53.244869 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:53.244835    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="589bae50-3c0e-466e-849b-cd6052804d65" path="/var/lib/kubelet/pods/589bae50-3c0e-466e-849b-cd6052804d65/volumes"
Nov 03 00:37:58.212175 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.212140    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"]
Nov 03 00:37:58.212883 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.212865    5012 cpu_manager.go:414] "containerMap: removing container" podUID="589bae50-3c0e-466e-849b-cd6052804d65" containerName="maas-api"
Nov 03 00:37:58.212883 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.212881    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="589bae50-3c0e-466e-849b-cd6052804d65" containerName="maas-api"
Nov 03 00:37:58.213097 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.213053    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="589bae50-3c0e-466e-849b-cd6052804d65" containerName="maas-api"
Nov 03 00:37:58.218384 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.218362    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.237404 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.237375    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"]
Nov 03 00:37:58.240786 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.240761    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-b43332-openshift-default-dockercfg-mvpsm\""
Nov 03 00:37:58.255503 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.255471    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-workload-socket\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.255638 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.255588    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84fbabc6-85b7-40f3-a12f-6182945af850-istiod-ca-cert\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.255726 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.255688    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84fbabc6-85b7-40f3-a12f-6182945af850-istio-token\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.255816 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.255737    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6992\" (UniqueName: \"kubernetes.io/projected/84fbabc6-85b7-40f3-a12f-6182945af850-kube-api-access-v6992\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.255899 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.255876    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-workload-certs\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.256032 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.256012    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-credential-socket\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.256147 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.256132    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-istio-data\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.256214 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.256177    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84fbabc6-85b7-40f3-a12f-6182945af850-istio-podinfo\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.256339 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.256317    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-istio-envoy\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.380957 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.370675    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-workload-socket\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.380957 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.370771    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84fbabc6-85b7-40f3-a12f-6182945af850-istiod-ca-cert\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.380957 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.370810    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84fbabc6-85b7-40f3-a12f-6182945af850-istio-token\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.380957 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.370847    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6992\" (UniqueName: \"kubernetes.io/projected/84fbabc6-85b7-40f3-a12f-6182945af850-kube-api-access-v6992\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.380957 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.377361    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-workload-socket\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.380957 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.378486    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84fbabc6-85b7-40f3-a12f-6182945af850-istiod-ca-cert\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.402012 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.392972    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-workload-certs\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.402012 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.393054    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-credential-socket\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.402012 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.393157    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-istio-data\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.402012 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.393184    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84fbabc6-85b7-40f3-a12f-6182945af850-istio-podinfo\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.402012 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.393216    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-istio-envoy\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.402012 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.398017    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-credential-socket\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.402012 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.398272    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-workload-certs\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.402012 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.398563    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-istio-data\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.415882 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.415302    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84fbabc6-85b7-40f3-a12f-6182945af850-istio-token\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.415882 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.415815    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84fbabc6-85b7-40f3-a12f-6182945af850-istio-podinfo\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.431472 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.430156    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-istio-envoy\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.431472 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.430455    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6992\" (UniqueName: \"kubernetes.io/projected/84fbabc6-85b7-40f3-a12f-6182945af850-kube-api-access-v6992\") pod \"e2e-explicit-b43332-openshift-default-5f447b8877-5k66w\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") " pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.539982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.537816    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:58.906122 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:37:58.906092    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84fbabc6_85b7_40f3_a12f_6182945af850.slice/crio-4b4081a33cb1eb54846073e206e665c6603b12aef183f4accfdffdc9fda9f6e2 WatchSource:0}: Error finding container 4b4081a33cb1eb54846073e206e665c6603b12aef183f4accfdffdc9fda9f6e2: Status 404 returned error can't find the container with id 4b4081a33cb1eb54846073e206e665c6603b12aef183f4accfdffdc9fda9f6e2
Nov 03 00:37:58.906690 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.906662    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"]
Nov 03 00:37:58.908713 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.908683    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:37:58.908788 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.908746    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:37:58.908788 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:58.908771    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:37:59.170722 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:59.170637    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w" event={"ID":"84fbabc6-85b7-40f3-a12f-6182945af850","Type":"ContainerStarted","Data":"50e8b67d6d66790c2f02b949a6ca1feaabe2ffda1797a996d126200466cb51ca"}
Nov 03 00:37:59.170722 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:59.170693    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w" event={"ID":"84fbabc6-85b7-40f3-a12f-6182945af850","Type":"ContainerStarted","Data":"4b4081a33cb1eb54846073e206e665c6603b12aef183f4accfdffdc9fda9f6e2"}
Nov 03 00:37:59.211967 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:59.211765    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w" podStartSLOduration=1.211747313 podStartE2EDuration="1.211747313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:37:59.207582791 +0000 UTC m=+1962.087166851" watchObservedRunningTime="2026-11-03 00:37:59.211747313 +0000 UTC m=+1962.091331355"
Nov 03 00:37:59.557687 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:59.557588    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:37:59.582719 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:37:59.580213    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:38:00.190022 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:00.189987    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:38:00.190879 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:00.190854    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:38:07.364846 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:38:07.364799    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:38:25.641309 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:25.641271    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8"]
Nov 03 00:38:25.647081 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:25.647061    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8"
Nov 03 00:38:25.665695 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:25.665666    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8"]
Nov 03 00:38:25.670464 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:25.670426    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-b43332-serving-cert\""
Nov 03 00:38:25.722161 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:25.722133    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68bc06b2-a513-46f8-adf9-c38b6689a05b-maas-api-tls\") pod \"maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8\" (UID: \"68bc06b2-a513-46f8-adf9-c38b6689a05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8"
Nov 03 00:38:25.722299 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:25.722273    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68bc06b2-a513-46f8-adf9-c38b6689a05b-metrics-certs\") pod \"maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8\" (UID: \"68bc06b2-a513-46f8-adf9-c38b6689a05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8"
Nov 03 00:38:25.722361 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:25.722315    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr79l\" (UniqueName: \"kubernetes.io/projected/68bc06b2-a513-46f8-adf9-c38b6689a05b-kube-api-access-fr79l\") pod \"maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8\" (UID: \"68bc06b2-a513-46f8-adf9-c38b6689a05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8"
Nov 03 00:38:25.722455 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:25.722423    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68bc06b2-a513-46f8-adf9-c38b6689a05b-openshift-service-ca\") pod \"maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8\" (UID: \"68bc06b2-a513-46f8-adf9-c38b6689a05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8"
Nov 03 00:38:25.823294 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:25.823270    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68bc06b2-a513-46f8-adf9-c38b6689a05b-maas-api-tls\") pod \"maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8\" (UID: \"68bc06b2-a513-46f8-adf9-c38b6689a05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8"
Nov 03 00:38:25.823448 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:25.823346    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68bc06b2-a513-46f8-adf9-c38b6689a05b-metrics-certs\") pod \"maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8\" (UID: \"68bc06b2-a513-46f8-adf9-c38b6689a05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8"
Nov 03 00:38:25.823448 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:25.823374    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr79l\" (UniqueName: \"kubernetes.io/projected/68bc06b2-a513-46f8-adf9-c38b6689a05b-kube-api-access-fr79l\") pod \"maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8\" (UID: \"68bc06b2-a513-46f8-adf9-c38b6689a05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8"
Nov 03 00:38:25.823448 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:25.823405    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68bc06b2-a513-46f8-adf9-c38b6689a05b-openshift-service-ca\") pod \"maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8\" (UID: \"68bc06b2-a513-46f8-adf9-c38b6689a05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8"
Nov 03 00:38:25.824108 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:25.824085    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68bc06b2-a513-46f8-adf9-c38b6689a05b-openshift-service-ca\") pod \"maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8\" (UID: \"68bc06b2-a513-46f8-adf9-c38b6689a05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8"
Nov 03 00:38:25.825902 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:25.825880    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68bc06b2-a513-46f8-adf9-c38b6689a05b-metrics-certs\") pod \"maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8\" (UID: \"68bc06b2-a513-46f8-adf9-c38b6689a05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8"
Nov 03 00:38:25.826056 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:25.825995    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68bc06b2-a513-46f8-adf9-c38b6689a05b-maas-api-tls\") pod \"maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8\" (UID: \"68bc06b2-a513-46f8-adf9-c38b6689a05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8"
Nov 03 00:38:25.875935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:25.847696    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr79l\" (UniqueName: \"kubernetes.io/projected/68bc06b2-a513-46f8-adf9-c38b6689a05b-kube-api-access-fr79l\") pod \"maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8\" (UID: \"68bc06b2-a513-46f8-adf9-c38b6689a05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8"
Nov 03 00:38:25.962845 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:25.962702    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8"
Nov 03 00:38:26.264074 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:38:26.264033    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68bc06b2_a513_46f8_adf9_c38b6689a05b.slice/crio-8d6c174b382d47c7ce9b84a00fadac248629f5b78afc90c01ad2188a0decf640 WatchSource:0}: Error finding container 8d6c174b382d47c7ce9b84a00fadac248629f5b78afc90c01ad2188a0decf640: Status 404 returned error can't find the container with id 8d6c174b382d47c7ce9b84a00fadac248629f5b78afc90c01ad2188a0decf640
Nov 03 00:38:26.264949 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:26.264921    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8"]
Nov 03 00:38:26.476366 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:26.476330    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8" event={"ID":"68bc06b2-a513-46f8-adf9-c38b6689a05b","Type":"ContainerStarted","Data":"8d6c174b382d47c7ce9b84a00fadac248629f5b78afc90c01ad2188a0decf640"}
Nov 03 00:38:27.487238 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:27.487201    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8" event={"ID":"68bc06b2-a513-46f8-adf9-c38b6689a05b","Type":"ContainerStarted","Data":"151fcb5929e86e0f525fb846812a584d2dd392cdfa87d4d04b56c23ab4897ffe"}
Nov 03 00:38:27.487627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:27.487409    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8"
Nov 03 00:38:27.541995 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:27.541945    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8" podStartSLOduration=2.215817975 podStartE2EDuration="2.541933303s" totalImagesPullingTime="326.115328ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:38:27.538276633 +0000 UTC m=+1990.417860675" watchObservedRunningTime="2026-11-03 00:38:27.541933303 +0000 UTC m=+1990.421517345"
Nov 03 00:38:33.502116 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:33.502081    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8"
Nov 03 00:38:37.364427 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:38:37.364393    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:38:45.027463 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:45.027427    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c247619c-mb256"]
Nov 03 00:38:45.037071 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:45.037043    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c247619c-mb256"]
Nov 03 00:38:45.245828 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:38:45.245793    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad3b283d-6e55-415e-b8b0-3210e403ab1a" path="/var/lib/kubelet/pods/ad3b283d-6e55-415e-b8b0-3210e403ab1a/volumes"
Nov 03 00:39:01.239936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:01.233108    5012 patch_prober.go:29] interesting pod/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:39:01.239936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:01.233185    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w" podUID="84fbabc6-85b7-40f3-a12f-6182945af850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:39:01.950212 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:01.950179    5012 generic.go:356] "Generic (PLEG): container finished" podID="84fbabc6-85b7-40f3-a12f-6182945af850" containerID="50e8b67d6d66790c2f02b949a6ca1feaabe2ffda1797a996d126200466cb51ca" exitCode=0
Nov 03 00:39:01.950401 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:01.950259    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w" event={"ID":"84fbabc6-85b7-40f3-a12f-6182945af850","Type":"ContainerDied","Data":"50e8b67d6d66790c2f02b949a6ca1feaabe2ffda1797a996d126200466cb51ca"}
Nov 03 00:39:01.950772 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:01.950750    5012 scope.go:122] "RemoveContainer" containerID="50e8b67d6d66790c2f02b949a6ca1feaabe2ffda1797a996d126200466cb51ca"
Nov 03 00:39:01.952682 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:01.952655    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:39:01.952760 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:01.952725    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:39:01.952800 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:01.952756    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:39:02.973818 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:02.973786    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w" event={"ID":"84fbabc6-85b7-40f3-a12f-6182945af850","Type":"ContainerStarted","Data":"86ac8d3a4ecc49eadbe07fa6d5b1c77e4c165bcfb90668f0bac6957b424a7be9"}
Nov 03 00:39:03.541069 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:03.541037    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:39:03.541244 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:03.541171    5012 patch_prober.go:29] interesting pod/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Nov 03 00:39:03.541244 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:03.541210    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w" podUID="84fbabc6-85b7-40f3-a12f-6182945af850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Nov 03 00:39:04.540798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:04.540760    5012 patch_prober.go:29] interesting pod/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Nov 03 00:39:04.541222 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:04.540818    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w" podUID="84fbabc6-85b7-40f3-a12f-6182945af850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Nov 03 00:39:05.541183 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:05.541144    5012 patch_prober.go:29] interesting pod/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Nov 03 00:39:05.541582 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:05.541208    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w" podUID="84fbabc6-85b7-40f3-a12f-6182945af850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Nov 03 00:39:06.540687 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:06.540653    5012 patch_prober.go:29] interesting pod/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Nov 03 00:39:06.540889 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:06.540722    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w" podUID="84fbabc6-85b7-40f3-a12f-6182945af850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Nov 03 00:39:07.366021 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:39:07.365987    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:39:07.541244 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:07.541211    5012 patch_prober.go:29] interesting pod/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Nov 03 00:39:07.541423 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:07.541263    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w" podUID="84fbabc6-85b7-40f3-a12f-6182945af850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Nov 03 00:39:08.544864 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:08.544831    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:39:08.545334 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:08.545314    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:39:08.546020 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:08.546002    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:39:37.364348 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:39:37.364313    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:39:38.028515 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:38.028474    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26c136dc-l88v7"]
Nov 03 00:39:38.037941 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:38.037899    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26c136dc-l88v7"]
Nov 03 00:39:39.246799 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:39.246766    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f598db0-3077-44f7-ba00-40ab57c0a64b" path="/var/lib/kubelet/pods/1f598db0-3077-44f7-ba00-40ab57c0a64b/volumes"
Nov 03 00:39:54.030888 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:54.030848    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9bb9669-q6jbv"]
Nov 03 00:39:54.040586 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:54.040558    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9bb9669-q6jbv"]
Nov 03 00:39:55.253562 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:39:55.253526    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41e7eae9-ad3b-4011-9f82-22e4436ceb79" path="/var/lib/kubelet/pods/41e7eae9-ad3b-4011-9f82-22e4436ceb79/volumes"
Nov 03 00:40:07.027742 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:07.027705    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d648ba4-9s9rs"]
Nov 03 00:40:07.037619 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:07.037594    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d648ba4-9s9rs"]
Nov 03 00:40:07.246251 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:07.246220    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82c48696-42c4-4710-beb9-434cfa1ed809" path="/var/lib/kubelet/pods/82c48696-42c4-4710-beb9-434cfa1ed809/volumes"
Nov 03 00:40:07.365266 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:07.365220    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:40:09.051739 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:09.051659    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8"]
Nov 03 00:40:09.052241 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:09.051945    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8" podUID="68bc06b2-a513-46f8-adf9-c38b6689a05b" containerName="maas-api" containerID="cri-o://151fcb5929e86e0f525fb846812a584d2dd392cdfa87d4d04b56c23ab4897ffe" gracePeriod=30
Nov 03 00:40:09.507972 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:09.502165    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8"
Nov 03 00:40:09.578728 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:09.578689    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68bc06b2-a513-46f8-adf9-c38b6689a05b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68bc06b2-a513-46f8-adf9-c38b6689a05b-maas-api-tls\") pod \"68bc06b2-a513-46f8-adf9-c38b6689a05b\" (UID: \"68bc06b2-a513-46f8-adf9-c38b6689a05b\") "
Nov 03 00:40:09.579262 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:09.579236    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68bc06b2-a513-46f8-adf9-c38b6689a05b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68bc06b2-a513-46f8-adf9-c38b6689a05b-openshift-service-ca\") pod \"68bc06b2-a513-46f8-adf9-c38b6689a05b\" (UID: \"68bc06b2-a513-46f8-adf9-c38b6689a05b\") "
Nov 03 00:40:09.579552 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:09.579536    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68bc06b2-a513-46f8-adf9-c38b6689a05b-kube-api-access-fr79l\" (UniqueName: \"kubernetes.io/projected/68bc06b2-a513-46f8-adf9-c38b6689a05b-kube-api-access-fr79l\") pod \"68bc06b2-a513-46f8-adf9-c38b6689a05b\" (UID: \"68bc06b2-a513-46f8-adf9-c38b6689a05b\") "
Nov 03 00:40:09.579727 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:09.579707    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68bc06b2-a513-46f8-adf9-c38b6689a05b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/68bc06b2-a513-46f8-adf9-c38b6689a05b-metrics-certs\") pod \"68bc06b2-a513-46f8-adf9-c38b6689a05b\" (UID: \"68bc06b2-a513-46f8-adf9-c38b6689a05b\") "
Nov 03 00:40:09.590926 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:09.590295    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68bc06b2-a513-46f8-adf9-c38b6689a05b-openshift-service-ca" pod "68bc06b2-a513-46f8-adf9-c38b6689a05b" (UID: "68bc06b2-a513-46f8-adf9-c38b6689a05b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:40:09.603940 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:09.595432    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68bc06b2-a513-46f8-adf9-c38b6689a05b-kube-api-access-fr79l" pod "68bc06b2-a513-46f8-adf9-c38b6689a05b" (UID: "68bc06b2-a513-46f8-adf9-c38b6689a05b"). InnerVolumeSpecName "kube-api-access-fr79l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:09.604104 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:09.604010    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68bc06b2-a513-46f8-adf9-c38b6689a05b-maas-api-tls" pod "68bc06b2-a513-46f8-adf9-c38b6689a05b" (UID: "68bc06b2-a513-46f8-adf9-c38b6689a05b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:40:09.610770 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:09.610741    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68bc06b2-a513-46f8-adf9-c38b6689a05b-metrics-certs" pod "68bc06b2-a513-46f8-adf9-c38b6689a05b" (UID: "68bc06b2-a513-46f8-adf9-c38b6689a05b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:40:09.682611 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:09.682569    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr79l\" (UniqueName: \"kubernetes.io/projected/68bc06b2-a513-46f8-adf9-c38b6689a05b-kube-api-access-fr79l\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:09.682611 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:09.682610    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68bc06b2-a513-46f8-adf9-c38b6689a05b-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:09.682809 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:09.682626    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68bc06b2-a513-46f8-adf9-c38b6689a05b-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:09.682809 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:09.682639    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68bc06b2-a513-46f8-adf9-c38b6689a05b-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:09.715379 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:09.715286    5012 generic.go:356] "Generic (PLEG): container finished" podID="68bc06b2-a513-46f8-adf9-c38b6689a05b" containerID="151fcb5929e86e0f525fb846812a584d2dd392cdfa87d4d04b56c23ab4897ffe" exitCode=0
Nov 03 00:40:09.715379 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:09.715329    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8"
Nov 03 00:40:09.715379 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:09.715357    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8" event={"ID":"68bc06b2-a513-46f8-adf9-c38b6689a05b","Type":"ContainerDied","Data":"151fcb5929e86e0f525fb846812a584d2dd392cdfa87d4d04b56c23ab4897ffe"}
Nov 03 00:40:09.715647 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:09.715428    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8" event={"ID":"68bc06b2-a513-46f8-adf9-c38b6689a05b","Type":"ContainerDied","Data":"8d6c174b382d47c7ce9b84a00fadac248629f5b78afc90c01ad2188a0decf640"}
Nov 03 00:40:09.715647 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:09.715463    5012 scope.go:122] "RemoveContainer" containerID="151fcb5929e86e0f525fb846812a584d2dd392cdfa87d4d04b56c23ab4897ffe"
Nov 03 00:40:09.727936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:09.727903    5012 scope.go:122] "RemoveContainer" containerID="151fcb5929e86e0f525fb846812a584d2dd392cdfa87d4d04b56c23ab4897ffe"
Nov 03 00:40:09.728264 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:09.728239    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"151fcb5929e86e0f525fb846812a584d2dd392cdfa87d4d04b56c23ab4897ffe\": container with ID starting with 151fcb5929e86e0f525fb846812a584d2dd392cdfa87d4d04b56c23ab4897ffe not found: ID does not exist" containerID="151fcb5929e86e0f525fb846812a584d2dd392cdfa87d4d04b56c23ab4897ffe"
Nov 03 00:40:09.728361 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:09.728269    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"151fcb5929e86e0f525fb846812a584d2dd392cdfa87d4d04b56c23ab4897ffe"} err="failed to get container status \"151fcb5929e86e0f525fb846812a584d2dd392cdfa87d4d04b56c23ab4897ffe\": rpc error: code = NotFound desc = could not find container \"151fcb5929e86e0f525fb846812a584d2dd392cdfa87d4d04b56c23ab4897ffe\": container with ID starting with 151fcb5929e86e0f525fb846812a584d2dd392cdfa87d4d04b56c23ab4897ffe not found: ID does not exist"
Nov 03 00:40:09.763192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:09.763166    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8"]
Nov 03 00:40:09.767214 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:09.767195    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b43332-7d8898dc74-dk6g8"]
Nov 03 00:40:11.244950 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:11.244897    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68bc06b2-a513-46f8-adf9-c38b6689a05b" path="/var/lib/kubelet/pods/68bc06b2-a513-46f8-adf9-c38b6689a05b/volumes"
Nov 03 00:40:17.323357 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:17.323320    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"]
Nov 03 00:40:17.323858 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:17.323563    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w" podUID="84fbabc6-85b7-40f3-a12f-6182945af850" containerName="istio-proxy" containerID="cri-o://86ac8d3a4ecc49eadbe07fa6d5b1c77e4c165bcfb90668f0bac6957b424a7be9" gracePeriod=30
Nov 03 00:40:18.094546 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:18.094520    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 00:40:18.099744 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:18.099720    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:40:18.109179 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:18.109153    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 00:40:18.113522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:18.113504    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:40:19.997863 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:19.997830    5012 scope.go:122] "RemoveContainer" containerID="9ed8649ce341881a66980839b1b144a8ad92a59e852e690745306b13c95825e1"
Nov 03 00:40:21.310247 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.310210    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"]
Nov 03 00:40:21.311070 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.311047    5012 cpu_manager.go:414] "containerMap: removing container" podUID="68bc06b2-a513-46f8-adf9-c38b6689a05b" containerName="maas-api"
Nov 03 00:40:21.311070 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.311068    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68bc06b2-a513-46f8-adf9-c38b6689a05b" containerName="maas-api"
Nov 03 00:40:21.311242 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.311173    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="68bc06b2-a513-46f8-adf9-c38b6689a05b" containerName="maas-api"
Nov 03 00:40:21.316668 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.316642    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.343710 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.343682    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-099583-openshift-default-dockercfg-hbtbr\""
Nov 03 00:40:21.347811 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.347782    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"]
Nov 03 00:40:21.392196 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.392169    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-credential-socket\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.392347 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.392232    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-envoy\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.392560 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.392541    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-workload-certs\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.392601 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.392588    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-data\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.392820 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.392797    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-workload-socket\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.392946 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.392919    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-podinfo\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.393030 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.392992    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8b85\" (UniqueName: \"kubernetes.io/projected/76d45dee-a04f-4d36-9496-6e6eef77b94a-kube-api-access-s8b85\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.393111 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.393094    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-token\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.393181 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.393166    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d45dee-a04f-4d36-9496-6e6eef77b94a-istiod-ca-cert\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.504933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.504841    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-envoy\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.505138 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.505100    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-workload-certs\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.505196 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.505164    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-data\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.505245 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.505232    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-workload-socket\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.508929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.505293    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-podinfo\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.508929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.505317    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8b85\" (UniqueName: \"kubernetes.io/projected/76d45dee-a04f-4d36-9496-6e6eef77b94a-kube-api-access-s8b85\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.508929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.505371    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-token\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.508929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.505407    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d45dee-a04f-4d36-9496-6e6eef77b94a-istiod-ca-cert\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.508929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.505449    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-credential-socket\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.508929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.505928    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-credential-socket\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.508929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.507179    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-data\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.508929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.507403    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-workload-certs\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.508929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.507607    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-workload-socket\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.527322 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.527261    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-envoy\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.547075 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.539673    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-token\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.547075 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.540762    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d45dee-a04f-4d36-9496-6e6eef77b94a-istiod-ca-cert\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.554952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.548704    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8b85\" (UniqueName: \"kubernetes.io/projected/76d45dee-a04f-4d36-9496-6e6eef77b94a-kube-api-access-s8b85\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.567660 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.567585    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-podinfo\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.637887 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.637840    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:21.823381 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.823295    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"]
Nov 03 00:40:21.823614 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.823562    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj" podUID="311cdd72-1e01-4590-ba0d-de18130c1d11" containerName="istio-proxy" containerID="cri-o://50a465e4bf5596de7c7e7d6695e06b1668a821b6f59792da0b83803f39799346" gracePeriod=30
Nov 03 00:40:21.827836 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.827813    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"]
Nov 03 00:40:21.828078 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.828016    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7" podUID="33ee11ba-035a-4f73-b194-12b3fdc9c32d" containerName="istio-proxy" containerID="cri-o://bcd2f45420a7b7ed278d3362edcbca6ed36d5d837848b7f3db4f7c554ba5b626" gracePeriod=30
Nov 03 00:40:21.990043 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:21.979525    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"]
Nov 03 00:40:22.023644 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:22.022703    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:40:22.023644 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:22.022797    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:40:22.023644 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:22.022841    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:40:22.872170 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:22.872133    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq" event={"ID":"76d45dee-a04f-4d36-9496-6e6eef77b94a","Type":"ContainerStarted","Data":"b643215a0a0279fff23cb45def4f49d25a7d531f12a10a1bf8f698788b783a32"}
Nov 03 00:40:22.872618 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:22.872200    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq" event={"ID":"76d45dee-a04f-4d36-9496-6e6eef77b94a","Type":"ContainerStarted","Data":"e4e149c98a29b7825d3635e1d387f01805c48fb2b5e4c000a6dd7e145e6c413d"}
Nov 03 00:40:22.874818 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:22.874788    5012 generic.go:356] "Generic (PLEG): container finished" podID="84fbabc6-85b7-40f3-a12f-6182945af850" containerID="86ac8d3a4ecc49eadbe07fa6d5b1c77e4c165bcfb90668f0bac6957b424a7be9" exitCode=0
Nov 03 00:40:22.874949 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:22.874853    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w" event={"ID":"84fbabc6-85b7-40f3-a12f-6182945af850","Type":"ContainerDied","Data":"86ac8d3a4ecc49eadbe07fa6d5b1c77e4c165bcfb90668f0bac6957b424a7be9"}
Nov 03 00:40:22.874949 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:22.874940    5012 scope.go:122] "RemoveContainer" containerID="50e8b67d6d66790c2f02b949a6ca1feaabe2ffda1797a996d126200466cb51ca"
Nov 03 00:40:22.937623 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:22.937552    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq" podStartSLOduration=1.937533815 podStartE2EDuration="1.937533815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:22.921495971 +0000 UTC m=+2105.801080043" watchObservedRunningTime="2026-11-03 00:40:22.937533815 +0000 UTC m=+2105.817117861"
Nov 03 00:40:23.021530 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.021502    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:40:23.140113 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.140083    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84fbabc6-85b7-40f3-a12f-6182945af850-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84fbabc6-85b7-40f3-a12f-6182945af850-istiod-ca-cert\") pod \"84fbabc6-85b7-40f3-a12f-6182945af850\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") "
Nov 03 00:40:23.140300 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.140200    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-workload-socket\") pod \"84fbabc6-85b7-40f3-a12f-6182945af850\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") "
Nov 03 00:40:23.140300 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.140224    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84fbabc6-85b7-40f3-a12f-6182945af850-istio-token\" (UniqueName: \"kubernetes.io/projected/84fbabc6-85b7-40f3-a12f-6182945af850-istio-token\") pod \"84fbabc6-85b7-40f3-a12f-6182945af850\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") "
Nov 03 00:40:23.140390 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.140296    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-istio-data\") pod \"84fbabc6-85b7-40f3-a12f-6182945af850\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") "
Nov 03 00:40:23.140441 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.140402    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84fbabc6-85b7-40f3-a12f-6182945af850-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84fbabc6-85b7-40f3-a12f-6182945af850-istio-podinfo\") pod \"84fbabc6-85b7-40f3-a12f-6182945af850\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") "
Nov 03 00:40:23.140494 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.140434    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84fbabc6-85b7-40f3-a12f-6182945af850-kube-api-access-v6992\" (UniqueName: \"kubernetes.io/projected/84fbabc6-85b7-40f3-a12f-6182945af850-kube-api-access-v6992\") pod \"84fbabc6-85b7-40f3-a12f-6182945af850\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") "
Nov 03 00:40:23.140548 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.140518    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-istio-envoy\") pod \"84fbabc6-85b7-40f3-a12f-6182945af850\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") "
Nov 03 00:40:23.140636 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.140553    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-workload-socket" pod "84fbabc6-85b7-40f3-a12f-6182945af850" (UID: "84fbabc6-85b7-40f3-a12f-6182945af850"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:23.140709 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.140622    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84fbabc6-85b7-40f3-a12f-6182945af850-istiod-ca-cert" pod "84fbabc6-85b7-40f3-a12f-6182945af850" (UID: "84fbabc6-85b7-40f3-a12f-6182945af850"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:40:23.140709 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.140640    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-istio-data" pod "84fbabc6-85b7-40f3-a12f-6182945af850" (UID: "84fbabc6-85b7-40f3-a12f-6182945af850"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:23.140709 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.140667    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-workload-certs\") pod \"84fbabc6-85b7-40f3-a12f-6182945af850\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") "
Nov 03 00:40:23.140877 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.140751    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-credential-socket\") pod \"84fbabc6-85b7-40f3-a12f-6182945af850\" (UID: \"84fbabc6-85b7-40f3-a12f-6182945af850\") "
Nov 03 00:40:23.141049 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.141019    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-workload-certs" pod "84fbabc6-85b7-40f3-a12f-6182945af850" (UID: "84fbabc6-85b7-40f3-a12f-6182945af850"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:23.141143 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.141028    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-credential-socket" pod "84fbabc6-85b7-40f3-a12f-6182945af850" (UID: "84fbabc6-85b7-40f3-a12f-6182945af850"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:23.141637 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.141613    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84fbabc6-85b7-40f3-a12f-6182945af850-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:23.141732 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.141644    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:23.141732 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.141658    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:23.141732 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.141673    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:23.141732 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.141686    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:23.142949 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.142899    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-istio-envoy" pod "84fbabc6-85b7-40f3-a12f-6182945af850" (UID: "84fbabc6-85b7-40f3-a12f-6182945af850"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:23.143048 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.142984    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84fbabc6-85b7-40f3-a12f-6182945af850-istio-podinfo" pod "84fbabc6-85b7-40f3-a12f-6182945af850" (UID: "84fbabc6-85b7-40f3-a12f-6182945af850"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:40:23.143179 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.143161    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84fbabc6-85b7-40f3-a12f-6182945af850-kube-api-access-v6992" pod "84fbabc6-85b7-40f3-a12f-6182945af850" (UID: "84fbabc6-85b7-40f3-a12f-6182945af850"). InnerVolumeSpecName "kube-api-access-v6992". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:23.144086 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.144070    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84fbabc6-85b7-40f3-a12f-6182945af850-istio-token" pod "84fbabc6-85b7-40f3-a12f-6182945af850" (UID: "84fbabc6-85b7-40f3-a12f-6182945af850"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:23.242501 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.242474    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84fbabc6-85b7-40f3-a12f-6182945af850-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:23.242501 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.242501    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84fbabc6-85b7-40f3-a12f-6182945af850-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:23.242656 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.242510    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6992\" (UniqueName: \"kubernetes.io/projected/84fbabc6-85b7-40f3-a12f-6182945af850-kube-api-access-v6992\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:23.242656 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.242519    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84fbabc6-85b7-40f3-a12f-6182945af850-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:23.638971 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.638936    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:23.645224 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.645200    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:23.886208 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.886180    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"
Nov 03 00:40:23.886748 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.886206    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w" event={"ID":"84fbabc6-85b7-40f3-a12f-6182945af850","Type":"ContainerDied","Data":"4b4081a33cb1eb54846073e206e665c6603b12aef183f4accfdffdc9fda9f6e2"}
Nov 03 00:40:23.886748 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.886272    5012 scope.go:122] "RemoveContainer" containerID="86ac8d3a4ecc49eadbe07fa6d5b1c77e4c165bcfb90668f0bac6957b424a7be9"
Nov 03 00:40:23.886872 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.886771    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:23.887890 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.887873    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:40:23.975542 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.974973    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"]
Nov 03 00:40:23.986848 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:23.986812    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-b43332-openshift-default-5f447b8877-5k66w"]
Nov 03 00:40:24.540954 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:24.535993    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"]
Nov 03 00:40:25.245616 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:25.245580    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84fbabc6-85b7-40f3-a12f-6182945af850" path="/var/lib/kubelet/pods/84fbabc6-85b7-40f3-a12f-6182945af850/volumes"
Nov 03 00:40:27.048503 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.048423    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575"]
Nov 03 00:40:27.049594 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.049566    5012 cpu_manager.go:414] "containerMap: removing container" podUID="84fbabc6-85b7-40f3-a12f-6182945af850" containerName="istio-proxy"
Nov 03 00:40:27.049594 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.049594    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84fbabc6-85b7-40f3-a12f-6182945af850" containerName="istio-proxy"
Nov 03 00:40:27.049860 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.049845    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="84fbabc6-85b7-40f3-a12f-6182945af850" containerName="istio-proxy"
Nov 03 00:40:27.049955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.049873    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="84fbabc6-85b7-40f3-a12f-6182945af850" containerName="istio-proxy"
Nov 03 00:40:27.054614 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.054588    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575"
Nov 03 00:40:27.073070 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.073043    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-099583\""
Nov 03 00:40:27.075592 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.075564    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575"]
Nov 03 00:40:27.137611 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.137581    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:40:27.242937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.240363    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-podinfo\") pod \"311cdd72-1e01-4590-ba0d-de18130c1d11\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") "
Nov 03 00:40:27.242937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.240455    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311cdd72-1e01-4590-ba0d-de18130c1d11-kube-api-access-25w9b\" (UniqueName: \"kubernetes.io/projected/311cdd72-1e01-4590-ba0d-de18130c1d11-kube-api-access-25w9b\") pod \"311cdd72-1e01-4590-ba0d-de18130c1d11\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") "
Nov 03 00:40:27.242937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.240555    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-workload-certs\") pod \"311cdd72-1e01-4590-ba0d-de18130c1d11\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") "
Nov 03 00:40:27.242937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.240603    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-envoy\") pod \"311cdd72-1e01-4590-ba0d-de18130c1d11\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") "
Nov 03 00:40:27.242937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.240836    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-workload-socket\") pod \"311cdd72-1e01-4590-ba0d-de18130c1d11\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") "
Nov 03 00:40:27.242937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.240883    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/311cdd72-1e01-4590-ba0d-de18130c1d11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311cdd72-1e01-4590-ba0d-de18130c1d11-istiod-ca-cert\") pod \"311cdd72-1e01-4590-ba0d-de18130c1d11\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") "
Nov 03 00:40:27.242937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.240942    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-data\") pod \"311cdd72-1e01-4590-ba0d-de18130c1d11\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") "
Nov 03 00:40:27.242937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.241041    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-token\" (UniqueName: \"kubernetes.io/projected/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-token\") pod \"311cdd72-1e01-4590-ba0d-de18130c1d11\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") "
Nov 03 00:40:27.242937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.241074    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-credential-socket\") pod \"311cdd72-1e01-4590-ba0d-de18130c1d11\" (UID: \"311cdd72-1e01-4590-ba0d-de18130c1d11\") "
Nov 03 00:40:27.242937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.241590    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e3bfaef-0df4-4b55-9eba-3761cf145de0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575\" (UID: \"7e3bfaef-0df4-4b55-9eba-3761cf145de0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575"
Nov 03 00:40:27.242937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.241683    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbflx\" (UniqueName: \"kubernetes.io/projected/7e3bfaef-0df4-4b55-9eba-3761cf145de0-kube-api-access-zbflx\") pod \"payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575\" (UID: \"7e3bfaef-0df4-4b55-9eba-3761cf145de0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575"
Nov 03 00:40:27.304972 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.296078    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-token" pod "311cdd72-1e01-4590-ba0d-de18130c1d11" (UID: "311cdd72-1e01-4590-ba0d-de18130c1d11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:27.304972 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.296273    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311cdd72-1e01-4590-ba0d-de18130c1d11-istiod-ca-cert" pod "311cdd72-1e01-4590-ba0d-de18130c1d11" (UID: "311cdd72-1e01-4590-ba0d-de18130c1d11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:40:27.321498 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.320366    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-data" pod "311cdd72-1e01-4590-ba0d-de18130c1d11" (UID: "311cdd72-1e01-4590-ba0d-de18130c1d11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:27.321498 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.320768    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-workload-certs" pod "311cdd72-1e01-4590-ba0d-de18130c1d11" (UID: "311cdd72-1e01-4590-ba0d-de18130c1d11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:27.321498 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.321021    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-credential-socket" pod "311cdd72-1e01-4590-ba0d-de18130c1d11" (UID: "311cdd72-1e01-4590-ba0d-de18130c1d11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:27.322232 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.321808    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-workload-socket" pod "311cdd72-1e01-4590-ba0d-de18130c1d11" (UID: "311cdd72-1e01-4590-ba0d-de18130c1d11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:27.429982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.410479    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e3bfaef-0df4-4b55-9eba-3761cf145de0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575\" (UID: \"7e3bfaef-0df4-4b55-9eba-3761cf145de0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575"
Nov 03 00:40:27.430444 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.430407    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbflx\" (UniqueName: \"kubernetes.io/projected/7e3bfaef-0df4-4b55-9eba-3761cf145de0-kube-api-access-zbflx\") pod \"payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575\" (UID: \"7e3bfaef-0df4-4b55-9eba-3761cf145de0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575"
Nov 03 00:40:27.431948 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.431901    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:27.432063 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.431958    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:27.432063 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.431974    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311cdd72-1e01-4590-ba0d-de18130c1d11-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:27.432063 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.431991    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:27.432063 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.432010    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:27.432063 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.432025    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:27.432063 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.419953    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e3bfaef-0df4-4b55-9eba-3761cf145de0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575\" (UID: \"7e3bfaef-0df4-4b55-9eba-3761cf145de0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575"
Nov 03 00:40:27.461748 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.461660    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:40:27.468496 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.468459    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-podinfo" pod "311cdd72-1e01-4590-ba0d-de18130c1d11" (UID: "311cdd72-1e01-4590-ba0d-de18130c1d11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:40:27.468639 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.468597    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311cdd72-1e01-4590-ba0d-de18130c1d11-kube-api-access-25w9b" pod "311cdd72-1e01-4590-ba0d-de18130c1d11" (UID: "311cdd72-1e01-4590-ba0d-de18130c1d11"). InnerVolumeSpecName "kube-api-access-25w9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:27.468724 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.468662    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-envoy" pod "311cdd72-1e01-4590-ba0d-de18130c1d11" (UID: "311cdd72-1e01-4590-ba0d-de18130c1d11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:27.470632 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.470600    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbflx\" (UniqueName: \"kubernetes.io/projected/7e3bfaef-0df4-4b55-9eba-3761cf145de0-kube-api-access-zbflx\") pod \"payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575\" (UID: \"7e3bfaef-0df4-4b55-9eba-3761cf145de0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575"
Nov 03 00:40:27.533272 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.533235    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istiod-ca-cert\") pod \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") "
Nov 03 00:40:27.533483 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.533353    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-envoy\") pod \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") "
Nov 03 00:40:27.533747 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.533715    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istiod-ca-cert" pod "33ee11ba-035a-4f73-b194-12b3fdc9c32d" (UID: "33ee11ba-035a-4f73-b194-12b3fdc9c32d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:40:27.533885 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.533864    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ee11ba-035a-4f73-b194-12b3fdc9c32d-kube-api-access-t79j2\" (UniqueName: \"kubernetes.io/projected/33ee11ba-035a-4f73-b194-12b3fdc9c32d-kube-api-access-t79j2\") pod \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") "
Nov 03 00:40:27.534028 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.534010    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-token\" (UniqueName: \"kubernetes.io/projected/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-token\") pod \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") "
Nov 03 00:40:27.534137 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.534093    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-workload-socket\") pod \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") "
Nov 03 00:40:27.534199 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.534175    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-workload-certs\") pod \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") "
Nov 03 00:40:27.534259 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.534232    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-podinfo\") pod \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") "
Nov 03 00:40:27.534311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.534267    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-credential-socket\") pod \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") "
Nov 03 00:40:27.534417 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.534400    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-data\") pod \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\" (UID: \"33ee11ba-035a-4f73-b194-12b3fdc9c32d\") "
Nov 03 00:40:27.534776 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.534751    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-workload-certs" pod "33ee11ba-035a-4f73-b194-12b3fdc9c32d" (UID: "33ee11ba-035a-4f73-b194-12b3fdc9c32d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:27.535281 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.535255    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:27.535735 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.535714    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:27.535735 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.535724    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-credential-socket" pod "33ee11ba-035a-4f73-b194-12b3fdc9c32d" (UID: "33ee11ba-035a-4f73-b194-12b3fdc9c32d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:27.535896 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.535736    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25w9b\" (UniqueName: \"kubernetes.io/projected/311cdd72-1e01-4590-ba0d-de18130c1d11-kube-api-access-25w9b\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:27.535896 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.535752    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311cdd72-1e01-4590-ba0d-de18130c1d11-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:27.535896 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.535765    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:27.536132 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.535981    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-data" pod "33ee11ba-035a-4f73-b194-12b3fdc9c32d" (UID: "33ee11ba-035a-4f73-b194-12b3fdc9c32d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:27.536132 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.536009    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-workload-socket" pod "33ee11ba-035a-4f73-b194-12b3fdc9c32d" (UID: "33ee11ba-035a-4f73-b194-12b3fdc9c32d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:27.536262 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.536192    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-envoy" pod "33ee11ba-035a-4f73-b194-12b3fdc9c32d" (UID: "33ee11ba-035a-4f73-b194-12b3fdc9c32d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:27.538067 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.538035    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ee11ba-035a-4f73-b194-12b3fdc9c32d-kube-api-access-t79j2" pod "33ee11ba-035a-4f73-b194-12b3fdc9c32d" (UID: "33ee11ba-035a-4f73-b194-12b3fdc9c32d"). InnerVolumeSpecName "kube-api-access-t79j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:27.538562 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.538542    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-token" pod "33ee11ba-035a-4f73-b194-12b3fdc9c32d" (UID: "33ee11ba-035a-4f73-b194-12b3fdc9c32d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:27.539138 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.539106    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-podinfo" pod "33ee11ba-035a-4f73-b194-12b3fdc9c32d" (UID: "33ee11ba-035a-4f73-b194-12b3fdc9c32d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:40:27.637403 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.637357    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:27.637403 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.637397    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:27.637654 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.637412    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:27.637654 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.637426    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t79j2\" (UniqueName: \"kubernetes.io/projected/33ee11ba-035a-4f73-b194-12b3fdc9c32d-kube-api-access-t79j2\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:27.637654 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.637439    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:27.637654 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.637452    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ee11ba-035a-4f73-b194-12b3fdc9c32d-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:27.637654 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.637467    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33ee11ba-035a-4f73-b194-12b3fdc9c32d-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:27.675989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.675889    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575"
Nov 03 00:40:27.926295 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.926259    5012 generic.go:356] "Generic (PLEG): container finished" podID="33ee11ba-035a-4f73-b194-12b3fdc9c32d" containerID="bcd2f45420a7b7ed278d3362edcbca6ed36d5d837848b7f3db4f7c554ba5b626" exitCode=0
Nov 03 00:40:27.926571 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.926359    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7" event={"ID":"33ee11ba-035a-4f73-b194-12b3fdc9c32d","Type":"ContainerDied","Data":"bcd2f45420a7b7ed278d3362edcbca6ed36d5d837848b7f3db4f7c554ba5b626"}
Nov 03 00:40:27.926760 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.926739    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7" event={"ID":"33ee11ba-035a-4f73-b194-12b3fdc9c32d","Type":"ContainerDied","Data":"93f4bd87e633f2ba2f40b58cbe53c9f866c7e8ebf170247768bb448ac97310d2"}
Nov 03 00:40:27.926882 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.926868    5012 scope.go:122] "RemoveContainer" containerID="bcd2f45420a7b7ed278d3362edcbca6ed36d5d837848b7f3db4f7c554ba5b626"
Nov 03 00:40:27.927111 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.926376    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"
Nov 03 00:40:27.929255 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.929226    5012 generic.go:356] "Generic (PLEG): container finished" podID="311cdd72-1e01-4590-ba0d-de18130c1d11" containerID="50a465e4bf5596de7c7e7d6695e06b1668a821b6f59792da0b83803f39799346" exitCode=0
Nov 03 00:40:27.929380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.929323    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj" event={"ID":"311cdd72-1e01-4590-ba0d-de18130c1d11","Type":"ContainerDied","Data":"50a465e4bf5596de7c7e7d6695e06b1668a821b6f59792da0b83803f39799346"}
Nov 03 00:40:27.929380 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.929365    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj" event={"ID":"311cdd72-1e01-4590-ba0d-de18130c1d11","Type":"ContainerDied","Data":"207b0d0e2fd2d96a80d81e863cca0289266bfc8c52af15b07db5a2c5efea3bbc"}
Nov 03 00:40:27.929632 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.929610    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"
Nov 03 00:40:27.949725 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.949553    5012 scope.go:122] "RemoveContainer" containerID="bcd2f45420a7b7ed278d3362edcbca6ed36d5d837848b7f3db4f7c554ba5b626"
Nov 03 00:40:27.949882 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:27.949863    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcd2f45420a7b7ed278d3362edcbca6ed36d5d837848b7f3db4f7c554ba5b626\": container with ID starting with bcd2f45420a7b7ed278d3362edcbca6ed36d5d837848b7f3db4f7c554ba5b626 not found: ID does not exist" containerID="bcd2f45420a7b7ed278d3362edcbca6ed36d5d837848b7f3db4f7c554ba5b626"
Nov 03 00:40:27.950064 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.949892    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcd2f45420a7b7ed278d3362edcbca6ed36d5d837848b7f3db4f7c554ba5b626"} err="failed to get container status \"bcd2f45420a7b7ed278d3362edcbca6ed36d5d837848b7f3db4f7c554ba5b626\": rpc error: code = NotFound desc = could not find container \"bcd2f45420a7b7ed278d3362edcbca6ed36d5d837848b7f3db4f7c554ba5b626\": container with ID starting with bcd2f45420a7b7ed278d3362edcbca6ed36d5d837848b7f3db4f7c554ba5b626 not found: ID does not exist"
Nov 03 00:40:27.950064 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.949926    5012 scope.go:122] "RemoveContainer" containerID="50a465e4bf5596de7c7e7d6695e06b1668a821b6f59792da0b83803f39799346"
Nov 03 00:40:27.960773 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.960753    5012 scope.go:122] "RemoveContainer" containerID="50a465e4bf5596de7c7e7d6695e06b1668a821b6f59792da0b83803f39799346"
Nov 03 00:40:27.961089 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:27.961070    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50a465e4bf5596de7c7e7d6695e06b1668a821b6f59792da0b83803f39799346\": container with ID starting with 50a465e4bf5596de7c7e7d6695e06b1668a821b6f59792da0b83803f39799346 not found: ID does not exist" containerID="50a465e4bf5596de7c7e7d6695e06b1668a821b6f59792da0b83803f39799346"
Nov 03 00:40:27.961151 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.961094    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50a465e4bf5596de7c7e7d6695e06b1668a821b6f59792da0b83803f39799346"} err="failed to get container status \"50a465e4bf5596de7c7e7d6695e06b1668a821b6f59792da0b83803f39799346\": rpc error: code = NotFound desc = could not find container \"50a465e4bf5596de7c7e7d6695e06b1668a821b6f59792da0b83803f39799346\": container with ID starting with 50a465e4bf5596de7c7e7d6695e06b1668a821b6f59792da0b83803f39799346 not found: ID does not exist"
Nov 03 00:40:27.987411 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.987389    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575"]
Nov 03 00:40:27.987613 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:40:27.987399    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3bfaef_0df4_4b55_9eba_3761cf145de0.slice/crio-64170a62fc576a66913cbf9bc977132375a08bcc669a58dd536a1f9bb5073fd9 WatchSource:0}: Error finding container 64170a62fc576a66913cbf9bc977132375a08bcc669a58dd536a1f9bb5073fd9: Status 404 returned error can't find the container with id 64170a62fc576a66913cbf9bc977132375a08bcc669a58dd536a1f9bb5073fd9
Nov 03 00:40:27.992708 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.992680    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"]
Nov 03 00:40:27.999294 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:27.998992    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zjzd7"]
Nov 03 00:40:28.015168 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:28.015139    5012 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ee11ba_035a_4f73_b194_12b3fdc9c32d.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311cdd72_1e01_4590_ba0d_de18130c1d11.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ee11ba_035a_4f73_b194_12b3fdc9c32d.slice/crio-93f4bd87e633f2ba2f40b58cbe53c9f866c7e8ebf170247768bb448ac97310d2\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311cdd72_1e01_4590_ba0d_de18130c1d11.slice/crio-207b0d0e2fd2d96a80d81e863cca0289266bfc8c52af15b07db5a2c5efea3bbc\": RecentStats: unable to find data in memory cache]"
Nov 03 00:40:28.023713 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:28.023689    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"]
Nov 03 00:40:28.032704 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:28.032680    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8pgtj"]
Nov 03 00:40:28.940955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:28.940903    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575" event={"ID":"7e3bfaef-0df4-4b55-9eba-3761cf145de0","Type":"ContainerStarted","Data":"dfe1afd709b554e9063ffa60322d4c39380f691e82d80e6ee472e1d86563ce68"}
Nov 03 00:40:28.941428 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:28.940971    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575" event={"ID":"7e3bfaef-0df4-4b55-9eba-3761cf145de0","Type":"ContainerStarted","Data":"64170a62fc576a66913cbf9bc977132375a08bcc669a58dd536a1f9bb5073fd9"}
Nov 03 00:40:28.941428 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:28.941023    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575"
Nov 03 00:40:28.979134 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:28.979075    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575" podStartSLOduration=1.9790568560000001 podStartE2EDuration="1.979056856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:28.977245839 +0000 UTC m=+2111.856829902" watchObservedRunningTime="2026-11-03 00:40:28.979056856 +0000 UTC m=+2111.858640900"
Nov 03 00:40:29.246872 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:29.246790    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="311cdd72-1e01-4590-ba0d-de18130c1d11" path="/var/lib/kubelet/pods/311cdd72-1e01-4590-ba0d-de18130c1d11/volumes"
Nov 03 00:40:29.247422 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:29.247405    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33ee11ba-035a-4f73-b194-12b3fdc9c32d" path="/var/lib/kubelet/pods/33ee11ba-035a-4f73-b194-12b3fdc9c32d/volumes"
Nov 03 00:40:30.909819 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:30.909775    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7612-18acc3"]
Nov 03 00:40:30.910745 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:30.910720    5012 cpu_manager.go:414] "containerMap: removing container" podUID="311cdd72-1e01-4590-ba0d-de18130c1d11" containerName="istio-proxy"
Nov 03 00:40:30.910745 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:30.910744    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="311cdd72-1e01-4590-ba0d-de18130c1d11" containerName="istio-proxy"
Nov 03 00:40:30.910966 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:30.910768    5012 cpu_manager.go:414] "containerMap: removing container" podUID="33ee11ba-035a-4f73-b194-12b3fdc9c32d" containerName="istio-proxy"
Nov 03 00:40:30.910966 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:30.910775    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33ee11ba-035a-4f73-b194-12b3fdc9c32d" containerName="istio-proxy"
Nov 03 00:40:30.910966 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:30.910791    5012 cpu_manager.go:414] "containerMap: removing container" podUID="84fbabc6-85b7-40f3-a12f-6182945af850" containerName="istio-proxy"
Nov 03 00:40:30.910966 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:30.910798    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84fbabc6-85b7-40f3-a12f-6182945af850" containerName="istio-proxy"
Nov 03 00:40:30.911192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:30.911029    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="311cdd72-1e01-4590-ba0d-de18130c1d11" containerName="istio-proxy"
Nov 03 00:40:30.911192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:30.911047    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="33ee11ba-035a-4f73-b194-12b3fdc9c32d" containerName="istio-proxy"
Nov 03 00:40:30.916169 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:30.916148    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-18acc3"
Nov 03 00:40:30.930592 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:30.930571    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-18acc3"]
Nov 03 00:40:30.961647 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:30.961626    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-b5fmb\""
Nov 03 00:40:31.002838 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:31.002810    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9kjk\" (UniqueName: \"kubernetes.io/projected/489829b6-f74c-450b-89a1-7656a6851566-kube-api-access-f9kjk\") pod \"test-curl-7612-18acc3\" (UID: \"489829b6-f74c-450b-89a1-7656a6851566\") " pod="opendatahub/test-curl-7612-18acc3"
Nov 03 00:40:31.104744 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:31.104719    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9kjk\" (UniqueName: \"kubernetes.io/projected/489829b6-f74c-450b-89a1-7656a6851566-kube-api-access-f9kjk\") pod \"test-curl-7612-18acc3\" (UID: \"489829b6-f74c-450b-89a1-7656a6851566\") " pod="opendatahub/test-curl-7612-18acc3"
Nov 03 00:40:31.122059 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:31.122036    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9kjk\" (UniqueName: \"kubernetes.io/projected/489829b6-f74c-450b-89a1-7656a6851566-kube-api-access-f9kjk\") pod \"test-curl-7612-18acc3\" (UID: \"489829b6-f74c-450b-89a1-7656a6851566\") " pod="opendatahub/test-curl-7612-18acc3"
Nov 03 00:40:31.230125 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:31.230062    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-18acc3"
Nov 03 00:40:31.539704 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:40:31.539672    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod489829b6_f74c_450b_89a1_7656a6851566.slice/crio-5c1a29ee3c72db76c737b3d56440620ede36264dcae1fcb5d5db458ef6fcb00d WatchSource:0}: Error finding container 5c1a29ee3c72db76c737b3d56440620ede36264dcae1fcb5d5db458ef6fcb00d: Status 404 returned error can't find the container with id 5c1a29ee3c72db76c737b3d56440620ede36264dcae1fcb5d5db458ef6fcb00d
Nov 03 00:40:31.539704 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:31.539683    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-18acc3"]
Nov 03 00:40:31.541786 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:31.541764    5012 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:40:31.967607 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:31.967572    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-18acc3" event={"ID":"489829b6-f74c-450b-89a1-7656a6851566","Type":"ContainerStarted","Data":"5c1a29ee3c72db76c737b3d56440620ede36264dcae1fcb5d5db458ef6fcb00d"}
Nov 03 00:40:32.978546 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:32.978452    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-18acc3" event={"ID":"489829b6-f74c-450b-89a1-7656a6851566","Type":"ContainerStarted","Data":"43c52afbeea9380d40bc2aaded7ce80fd6c8097a4f60950814acce1edb73f4ce"}
Nov 03 00:40:33.015088 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:33.015045    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7612-18acc3" podStartSLOduration=1.866495289 podStartE2EDuration="3.015031797s" totalImagesPullingTime="1.148536508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:33.011481912 +0000 UTC m=+2115.891065955" watchObservedRunningTime="2026-11-03 00:40:33.015031797 +0000 UTC m=+2115.894615839"
Nov 03 00:40:33.860010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:33.859976    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7612-18acc3"]
Nov 03 00:40:33.867068 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:33.867038    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7612-18acc3"]
Nov 03 00:40:34.009111 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:34.009081    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7612-c144dd"]
Nov 03 00:40:34.009814 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:34.009799    5012 cpu_manager.go:414] "containerMap: removing container" podUID="489829b6-f74c-450b-89a1-7656a6851566" containerName="test-curl-7612-18acc3"
Nov 03 00:40:34.009860 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:34.009814    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="489829b6-f74c-450b-89a1-7656a6851566" containerName="test-curl-7612-18acc3"
Nov 03 00:40:34.009941 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:34.009931    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="489829b6-f74c-450b-89a1-7656a6851566" containerName="test-curl-7612-18acc3"
Nov 03 00:40:34.013732 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:34.013715    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-c144dd"
Nov 03 00:40:34.028649 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:34.028625    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-c144dd"]
Nov 03 00:40:34.049391 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:34.049360    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzznk\" (UniqueName: \"kubernetes.io/projected/e844bf55-5492-47c9-95ea-4eda5c542f34-kube-api-access-fzznk\") pod \"test-curl-7612-c144dd\" (UID: \"e844bf55-5492-47c9-95ea-4eda5c542f34\") " pod="opendatahub/test-curl-7612-c144dd"
Nov 03 00:40:34.151144 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:34.151079    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzznk\" (UniqueName: \"kubernetes.io/projected/e844bf55-5492-47c9-95ea-4eda5c542f34-kube-api-access-fzznk\") pod \"test-curl-7612-c144dd\" (UID: \"e844bf55-5492-47c9-95ea-4eda5c542f34\") " pod="opendatahub/test-curl-7612-c144dd"
Nov 03 00:40:34.168838 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:34.168807    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzznk\" (UniqueName: \"kubernetes.io/projected/e844bf55-5492-47c9-95ea-4eda5c542f34-kube-api-access-fzznk\") pod \"test-curl-7612-c144dd\" (UID: \"e844bf55-5492-47c9-95ea-4eda5c542f34\") " pod="opendatahub/test-curl-7612-c144dd"
Nov 03 00:40:34.327884 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:34.327854    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-c144dd"
Nov 03 00:40:34.628065 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:40:34.628036    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode844bf55_5492_47c9_95ea_4eda5c542f34.slice/crio-90354456236b65eb32039abe6af000ca0f0310f7881c29726bbad3ca9a40e6d9 WatchSource:0}: Error finding container 90354456236b65eb32039abe6af000ca0f0310f7881c29726bbad3ca9a40e6d9: Status 404 returned error can't find the container with id 90354456236b65eb32039abe6af000ca0f0310f7881c29726bbad3ca9a40e6d9
Nov 03 00:40:34.628331 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:34.628307    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-c144dd"]
Nov 03 00:40:35.039800 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:35.039673    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-c144dd" event={"ID":"e844bf55-5492-47c9-95ea-4eda5c542f34","Type":"ContainerStarted","Data":"e404d45209e015374bb3c37f33b8f999f6d9feec9236129be8fdd047a45ccd3d"}
Nov 03 00:40:35.039800 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:35.039721    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-c144dd" event={"ID":"e844bf55-5492-47c9-95ea-4eda5c542f34","Type":"ContainerStarted","Data":"90354456236b65eb32039abe6af000ca0f0310f7881c29726bbad3ca9a40e6d9"}
Nov 03 00:40:35.040332 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:35.040037    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7612-18acc3" podUID="489829b6-f74c-450b-89a1-7656a6851566" containerName="test-curl-7612-18acc3" containerID="cri-o://43c52afbeea9380d40bc2aaded7ce80fd6c8097a4f60950814acce1edb73f4ce" gracePeriod=2
Nov 03 00:40:35.075414 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:35.075370    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7612-c144dd" podStartSLOduration=2.075355653 podStartE2EDuration="2.075355653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:35.073924815 +0000 UTC m=+2117.953508856" watchObservedRunningTime="2026-11-03 00:40:35.075355653 +0000 UTC m=+2117.954939696"
Nov 03 00:40:35.086193 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:35.086166    5012 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7612-18acc3\" is forbidden: User \"system:node:ip-10-0-133-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-161.ec2.internal' and this object" podUID="489829b6-f74c-450b-89a1-7656a6851566" pod="opendatahub/test-curl-7612-18acc3"
Nov 03 00:40:35.902046 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:35.902012    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7612-c144dd"]
Nov 03 00:40:35.907620 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:35.907592    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7612-c144dd"]
Nov 03 00:40:36.046279 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:36.046243    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7612-347ab8"]
Nov 03 00:40:36.047086 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:36.047067    5012 cpu_manager.go:414] "containerMap: removing container" podUID="e844bf55-5492-47c9-95ea-4eda5c542f34" containerName="test-curl-7612-c144dd"
Nov 03 00:40:36.047086 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:36.047086    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e844bf55-5492-47c9-95ea-4eda5c542f34" containerName="test-curl-7612-c144dd"
Nov 03 00:40:36.047215 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:36.047205    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="e844bf55-5492-47c9-95ea-4eda5c542f34" containerName="test-curl-7612-c144dd"
Nov 03 00:40:36.050977 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:36.050959    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-347ab8"
Nov 03 00:40:36.065035 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:36.065009    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-347ab8"]
Nov 03 00:40:36.177386 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:36.177316    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j587r\" (UniqueName: \"kubernetes.io/projected/b802e61b-eb07-4b2e-9306-18fdeb20e538-kube-api-access-j587r\") pod \"test-curl-7612-347ab8\" (UID: \"b802e61b-eb07-4b2e-9306-18fdeb20e538\") " pod="opendatahub/test-curl-7612-347ab8"
Nov 03 00:40:36.278213 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:36.278177    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j587r\" (UniqueName: \"kubernetes.io/projected/b802e61b-eb07-4b2e-9306-18fdeb20e538-kube-api-access-j587r\") pod \"test-curl-7612-347ab8\" (UID: \"b802e61b-eb07-4b2e-9306-18fdeb20e538\") " pod="opendatahub/test-curl-7612-347ab8"
Nov 03 00:40:36.295193 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:36.295169    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j587r\" (UniqueName: \"kubernetes.io/projected/b802e61b-eb07-4b2e-9306-18fdeb20e538-kube-api-access-j587r\") pod \"test-curl-7612-347ab8\" (UID: \"b802e61b-eb07-4b2e-9306-18fdeb20e538\") " pod="opendatahub/test-curl-7612-347ab8"
Nov 03 00:40:36.364836 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:36.364812    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-347ab8"
Nov 03 00:40:36.663449 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:36.663418    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-347ab8"]
Nov 03 00:40:36.663791 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:40:36.663762    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb802e61b_eb07_4b2e_9306_18fdeb20e538.slice/crio-0f081554e00debee8ef31ae7c8d6f01b501a61980c55b8e4436917fdc9b572c8 WatchSource:0}: Error finding container 0f081554e00debee8ef31ae7c8d6f01b501a61980c55b8e4436917fdc9b572c8: Status 404 returned error can't find the container with id 0f081554e00debee8ef31ae7c8d6f01b501a61980c55b8e4436917fdc9b572c8
Nov 03 00:40:36.960432 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:36.960355    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"]
Nov 03 00:40:36.960610 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:36.960586    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8" podUID="e42db167-69a8-4280-b3d6-3ac4c6a6a42f" containerName="istio-proxy" containerID="cri-o://a3d5ebfb03f5055a0b57d6dce75f69ceb3cbe3464d4bd2eb8f781415028f0149" gracePeriod=30
Nov 03 00:40:36.971281 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:36.971216    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"]
Nov 03 00:40:36.971490 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:36.971463    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx" podUID="2431e1a4-b1df-4516-a2bf-680be875a7d9" containerName="istio-proxy" containerID="cri-o://f01966f2449ee2cf6432e3e2285e05ba68cea6c1765bdd5a4b54236337e0563c" gracePeriod=30
Nov 03 00:40:36.976498 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:36.976470    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"]
Nov 03 00:40:36.981660 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:36.981632    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"]
Nov 03 00:40:36.981871 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:36.981847    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx" podUID="e9016037-dc28-41df-a3ad-5672f6f5f0c0" containerName="istio-proxy" containerID="cri-o://da45fb8997b49d1b9550527be582e6b1ff58c732a4519053c19bd0badba2fd6a" gracePeriod=30
Nov 03 00:40:37.059125 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:37.059095    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-347ab8" event={"ID":"b802e61b-eb07-4b2e-9306-18fdeb20e538","Type":"ContainerStarted","Data":"7e01a2d4a0b53aeaf521f15879e520fba382bae438cccd78dab812693c5d6904"}
Nov 03 00:40:37.059428 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:37.059135    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-347ab8" event={"ID":"b802e61b-eb07-4b2e-9306-18fdeb20e538","Type":"ContainerStarted","Data":"0f081554e00debee8ef31ae7c8d6f01b501a61980c55b8e4436917fdc9b572c8"}
Nov 03 00:40:37.059428 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:37.059356    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7612-c144dd" podUID="e844bf55-5492-47c9-95ea-4eda5c542f34" containerName="test-curl-7612-c144dd" containerID="cri-o://e404d45209e015374bb3c37f33b8f999f6d9feec9236129be8fdd047a45ccd3d" gracePeriod=2
Nov 03 00:40:37.059527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:37.059464    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p" podUID="7d9034f1-0b7c-4473-93be-4d859fb850a0" containerName="istio-proxy" containerID="cri-o://d13823617dc62bba559232a671af1bcb28e9af1518397d542be1a2d9e57342a3" gracePeriod=30
Nov 03 00:40:37.135926 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:37.135865    5012 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7612-c144dd\" is forbidden: User \"system:node:ip-10-0-133-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-161.ec2.internal' and this object" podUID="e844bf55-5492-47c9-95ea-4eda5c542f34" pod="opendatahub/test-curl-7612-c144dd"
Nov 03 00:40:37.169331 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:37.169280    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7612-347ab8" podStartSLOduration=1.16926644 podStartE2EDuration="1.16926644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:37.168548239 +0000 UTC m=+2120.048132283" watchObservedRunningTime="2026-11-03 00:40:37.16926644 +0000 UTC m=+2120.048850482"
Nov 03 00:40:37.252740 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:37.252709    5012 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7612-c144dd\" is forbidden: User \"system:node:ip-10-0-133-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-161.ec2.internal' and this object" podUID="e844bf55-5492-47c9-95ea-4eda5c542f34" pod="opendatahub/test-curl-7612-c144dd"
Nov 03 00:40:37.254769 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:37.254739    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-18acc3"
Nov 03 00:40:37.266431 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:37.266407    5012 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7612-18acc3\" is forbidden: User \"system:node:ip-10-0-133-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-161.ec2.internal' and this object" podUID="489829b6-f74c-450b-89a1-7656a6851566" pod="opendatahub/test-curl-7612-18acc3"
Nov 03 00:40:37.364158 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:37.364131    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:40:37.400895 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:37.400864    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/489829b6-f74c-450b-89a1-7656a6851566-kube-api-access-f9kjk\" (UniqueName: \"kubernetes.io/projected/489829b6-f74c-450b-89a1-7656a6851566-kube-api-access-f9kjk\") pod \"489829b6-f74c-450b-89a1-7656a6851566\" (UID: \"489829b6-f74c-450b-89a1-7656a6851566\") "
Nov 03 00:40:37.403397 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:37.403369    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/489829b6-f74c-450b-89a1-7656a6851566-kube-api-access-f9kjk" pod "489829b6-f74c-450b-89a1-7656a6851566" (UID: "489829b6-f74c-450b-89a1-7656a6851566"). InnerVolumeSpecName "kube-api-access-f9kjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:37.502721 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:37.502656    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9kjk\" (UniqueName: \"kubernetes.io/projected/489829b6-f74c-450b-89a1-7656a6851566-kube-api-access-f9kjk\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:37.982891 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:37.982855    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7612-347ab8"]
Nov 03 00:40:37.987727 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:37.987700    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7612-347ab8"]
Nov 03 00:40:38.068623 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:38.068584    5012 generic.go:356] "Generic (PLEG): container finished" podID="489829b6-f74c-450b-89a1-7656a6851566" containerID="43c52afbeea9380d40bc2aaded7ce80fd6c8097a4f60950814acce1edb73f4ce" exitCode=137
Nov 03 00:40:38.069094 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:38.068629    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-18acc3"
Nov 03 00:40:38.069094 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:38.068673    5012 scope.go:122] "RemoveContainer" containerID="43c52afbeea9380d40bc2aaded7ce80fd6c8097a4f60950814acce1edb73f4ce"
Nov 03 00:40:38.083610 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:38.083551    5012 scope.go:122] "RemoveContainer" containerID="43c52afbeea9380d40bc2aaded7ce80fd6c8097a4f60950814acce1edb73f4ce"
Nov 03 00:40:38.083698 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:38.083668    5012 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7612-18acc3\" is forbidden: User \"system:node:ip-10-0-133-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-161.ec2.internal' and this object" podUID="489829b6-f74c-450b-89a1-7656a6851566" pod="opendatahub/test-curl-7612-18acc3"
Nov 03 00:40:38.083878 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:38.083849    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43c52afbeea9380d40bc2aaded7ce80fd6c8097a4f60950814acce1edb73f4ce\": container with ID starting with 43c52afbeea9380d40bc2aaded7ce80fd6c8097a4f60950814acce1edb73f4ce not found: ID does not exist" containerID="43c52afbeea9380d40bc2aaded7ce80fd6c8097a4f60950814acce1edb73f4ce"
Nov 03 00:40:38.083986 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:38.083889    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43c52afbeea9380d40bc2aaded7ce80fd6c8097a4f60950814acce1edb73f4ce"} err="failed to get container status \"43c52afbeea9380d40bc2aaded7ce80fd6c8097a4f60950814acce1edb73f4ce\": rpc error: code = NotFound desc = could not find container \"43c52afbeea9380d40bc2aaded7ce80fd6c8097a4f60950814acce1edb73f4ce\": container with ID starting with 43c52afbeea9380d40bc2aaded7ce80fd6c8097a4f60950814acce1edb73f4ce not found: ID does not exist"
Nov 03 00:40:38.095260 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:38.095236    5012 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7612-18acc3\" is forbidden: User \"system:node:ip-10-0-133-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-161.ec2.internal' and this object" podUID="489829b6-f74c-450b-89a1-7656a6851566" pod="opendatahub/test-curl-7612-18acc3"
Nov 03 00:40:38.138157 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:38.138138    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7612-cca854"]
Nov 03 00:40:38.138812 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:38.138798    5012 cpu_manager.go:414] "containerMap: removing container" podUID="b802e61b-eb07-4b2e-9306-18fdeb20e538" containerName="test-curl-7612-347ab8"
Nov 03 00:40:38.138852 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:38.138813    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b802e61b-eb07-4b2e-9306-18fdeb20e538" containerName="test-curl-7612-347ab8"
Nov 03 00:40:38.138968 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:38.138956    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="b802e61b-eb07-4b2e-9306-18fdeb20e538" containerName="test-curl-7612-347ab8"
Nov 03 00:40:38.144001 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:38.143986    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-cca854"
Nov 03 00:40:38.155629 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:38.155597    5012 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7612-18acc3\" is forbidden: User \"system:node:ip-10-0-133-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-161.ec2.internal' and this object" podUID="489829b6-f74c-450b-89a1-7656a6851566" pod="opendatahub/test-curl-7612-18acc3"
Nov 03 00:40:38.158586 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:38.158561    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-cca854"]
Nov 03 00:40:38.210560 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:38.210526    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frwjm\" (UniqueName: \"kubernetes.io/projected/7b08c7f4-0fdc-4e18-a49d-b3ad1c79adea-kube-api-access-frwjm\") pod \"test-curl-7612-cca854\" (UID: \"7b08c7f4-0fdc-4e18-a49d-b3ad1c79adea\") " pod="opendatahub/test-curl-7612-cca854"
Nov 03 00:40:38.313985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:38.312687    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frwjm\" (UniqueName: \"kubernetes.io/projected/7b08c7f4-0fdc-4e18-a49d-b3ad1c79adea-kube-api-access-frwjm\") pod \"test-curl-7612-cca854\" (UID: \"7b08c7f4-0fdc-4e18-a49d-b3ad1c79adea\") " pod="opendatahub/test-curl-7612-cca854"
Nov 03 00:40:38.332398 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:38.332357    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frwjm\" (UniqueName: \"kubernetes.io/projected/7b08c7f4-0fdc-4e18-a49d-b3ad1c79adea-kube-api-access-frwjm\") pod \"test-curl-7612-cca854\" (UID: \"7b08c7f4-0fdc-4e18-a49d-b3ad1c79adea\") " pod="opendatahub/test-curl-7612-cca854"
Nov 03 00:40:38.457407 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:38.457376    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-cca854"
Nov 03 00:40:38.757983 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:38.757956    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-cca854"]
Nov 03 00:40:38.758625 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:40:38.758601    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b08c7f4_0fdc_4e18_a49d_b3ad1c79adea.slice/crio-c59f584d3eb3c5ff53fa15ff5a5ba695c91682386724c4dfe06aa759a14b82f8 WatchSource:0}: Error finding container c59f584d3eb3c5ff53fa15ff5a5ba695c91682386724c4dfe06aa759a14b82f8: Status 404 returned error can't find the container with id c59f584d3eb3c5ff53fa15ff5a5ba695c91682386724c4dfe06aa759a14b82f8
Nov 03 00:40:39.078657 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.078621    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-cca854" event={"ID":"7b08c7f4-0fdc-4e18-a49d-b3ad1c79adea","Type":"ContainerStarted","Data":"27ed6bd714b681610a85259ab4e2c1fe3e87e7d0589cc22e62c93fb1669e1cd5"}
Nov 03 00:40:39.079023 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.078676    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-cca854" event={"ID":"7b08c7f4-0fdc-4e18-a49d-b3ad1c79adea","Type":"ContainerStarted","Data":"c59f584d3eb3c5ff53fa15ff5a5ba695c91682386724c4dfe06aa759a14b82f8"}
Nov 03 00:40:39.079949 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.079884    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7612-347ab8" podUID="b802e61b-eb07-4b2e-9306-18fdeb20e538" containerName="test-curl-7612-347ab8" containerID="cri-o://7e01a2d4a0b53aeaf521f15879e520fba382bae438cccd78dab812693c5d6904" gracePeriod=2
Nov 03 00:40:39.112594 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.112543    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7612-cca854" podStartSLOduration=1.112528152 podStartE2EDuration="1.112528152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:39.112208329 +0000 UTC m=+2121.991792364" watchObservedRunningTime="2026-11-03 00:40:39.112528152 +0000 UTC m=+2121.992112199"
Nov 03 00:40:39.123895 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:39.123860    5012 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7612-18acc3\" is forbidden: User \"system:node:ip-10-0-133-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-161.ec2.internal' and this object" podUID="489829b6-f74c-450b-89a1-7656a6851566" pod="opendatahub/test-curl-7612-18acc3"
Nov 03 00:40:39.135647 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:39.135603    5012 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7612-18acc3\" is forbidden: User \"system:node:ip-10-0-133-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-161.ec2.internal' and this object" podUID="489829b6-f74c-450b-89a1-7656a6851566" pod="opendatahub/test-curl-7612-18acc3"
Nov 03 00:40:39.147016 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:39.146987    5012 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7612-347ab8\" is forbidden: User \"system:node:ip-10-0-133-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-161.ec2.internal' and this object" podUID="b802e61b-eb07-4b2e-9306-18fdeb20e538" pod="opendatahub/test-curl-7612-347ab8"
Nov 03 00:40:39.246432 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.246399    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="489829b6-f74c-450b-89a1-7656a6851566" path="/var/lib/kubelet/pods/489829b6-f74c-450b-89a1-7656a6851566/volumes"
Nov 03 00:40:39.252076 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.252056    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-c144dd"
Nov 03 00:40:39.327762 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.327692    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e844bf55-5492-47c9-95ea-4eda5c542f34-kube-api-access-fzznk\" (UniqueName: \"kubernetes.io/projected/e844bf55-5492-47c9-95ea-4eda5c542f34-kube-api-access-fzznk\") pod \"e844bf55-5492-47c9-95ea-4eda5c542f34\" (UID: \"e844bf55-5492-47c9-95ea-4eda5c542f34\") "
Nov 03 00:40:39.329824 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.329799    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e844bf55-5492-47c9-95ea-4eda5c542f34-kube-api-access-fzznk" pod "e844bf55-5492-47c9-95ea-4eda5c542f34" (UID: "e844bf55-5492-47c9-95ea-4eda5c542f34"). InnerVolumeSpecName "kube-api-access-fzznk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:39.429622 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.429595    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzznk\" (UniqueName: \"kubernetes.io/projected/e844bf55-5492-47c9-95ea-4eda5c542f34-kube-api-access-fzznk\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:39.558866 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.558833    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g" podUID="bc53fb3c-4315-4804-b73e-c4c44c9fbca7" containerName="main" containerID="cri-o://818fd94c77d6023e3127ed72fcc9fad31a132ff443bcdc7c949c2fb10f5508d7" gracePeriod=45
Nov 03 00:40:39.856139 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.856110    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:40:39.937100 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.937061    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-dshm\") pod \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\" (UID: \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\") "
Nov 03 00:40:39.937283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.937142    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-tmp-dir\") pod \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\" (UID: \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\") "
Nov 03 00:40:39.937283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.937196    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-kube-api-access-6sdxv\" (UniqueName: \"kubernetes.io/projected/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-kube-api-access-6sdxv\") pod \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\" (UID: \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\") "
Nov 03 00:40:39.937283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.937258    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-home\" (UniqueName: \"kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-home\") pod \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\" (UID: \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\") "
Nov 03 00:40:39.937534 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.937505    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-tmp-dir" pod "bc53fb3c-4315-4804-b73e-c4c44c9fbca7" (UID: "bc53fb3c-4315-4804-b73e-c4c44c9fbca7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:39.937619 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.937518    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-home" pod "bc53fb3c-4315-4804-b73e-c4c44c9fbca7" (UID: "bc53fb3c-4315-4804-b73e-c4c44c9fbca7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:39.937619 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.937543    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-model-cache\") pod \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\" (UID: \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\") "
Nov 03 00:40:39.937743 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.937620    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-tls-certs\" (UniqueName: \"kubernetes.io/secret/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-tls-certs\") pod \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\" (UID: \"bc53fb3c-4315-4804-b73e-c4c44c9fbca7\") "
Nov 03 00:40:39.937805 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.937741    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-model-cache" pod "bc53fb3c-4315-4804-b73e-c4c44c9fbca7" (UID: "bc53fb3c-4315-4804-b73e-c4c44c9fbca7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:39.938549 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.938524    5012 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-tmp-dir\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:39.938682 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.938552    5012 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-home\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:39.938682 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.938567    5012 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-model-cache\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:39.939798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.939768    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-dshm" pod "bc53fb3c-4315-4804-b73e-c4c44c9fbca7" (UID: "bc53fb3c-4315-4804-b73e-c4c44c9fbca7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:39.939943 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.939810    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-kube-api-access-6sdxv" pod "bc53fb3c-4315-4804-b73e-c4c44c9fbca7" (UID: "bc53fb3c-4315-4804-b73e-c4c44c9fbca7"). InnerVolumeSpecName "kube-api-access-6sdxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:39.940197 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:39.940177    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-tls-certs" pod "bc53fb3c-4315-4804-b73e-c4c44c9fbca7" (UID: "bc53fb3c-4315-4804-b73e-c4c44c9fbca7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:40:40.009932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:40.007984    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7612-cca854"]
Nov 03 00:40:40.014754 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:40.013841    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575"
Nov 03 00:40:40.034672 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:40.034639    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7612-cca854"]
Nov 03 00:40:40.040335 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:40.040309    5012 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-dshm\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:40.040335 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:40.040336    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sdxv\" (UniqueName: \"kubernetes.io/projected/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-kube-api-access-6sdxv\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:40.040490 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:40.040345    5012 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc53fb3c-4315-4804-b73e-c4c44c9fbca7-tls-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:40.091696 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:40.091618    5012 generic.go:356] "Generic (PLEG): container finished" podID="bc53fb3c-4315-4804-b73e-c4c44c9fbca7" containerID="818fd94c77d6023e3127ed72fcc9fad31a132ff443bcdc7c949c2fb10f5508d7" exitCode=0
Nov 03 00:40:40.091696 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:40.091682    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"
Nov 03 00:40:40.092224 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:40.091771    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g" event={"ID":"bc53fb3c-4315-4804-b73e-c4c44c9fbca7","Type":"ContainerDied","Data":"818fd94c77d6023e3127ed72fcc9fad31a132ff443bcdc7c949c2fb10f5508d7"}
Nov 03 00:40:40.092224 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:40.091808    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g" event={"ID":"bc53fb3c-4315-4804-b73e-c4c44c9fbca7","Type":"ContainerDied","Data":"b95a823e8fb391d368f2b5dadec93fba6cc91d882ca85f437b32e0ce1d04484c"}
Nov 03 00:40:40.092224 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:40.091833    5012 scope.go:122] "RemoveContainer" containerID="818fd94c77d6023e3127ed72fcc9fad31a132ff443bcdc7c949c2fb10f5508d7"
Nov 03 00:40:40.096503 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:40.096478    5012 generic.go:356] "Generic (PLEG): container finished" podID="e844bf55-5492-47c9-95ea-4eda5c542f34" containerID="e404d45209e015374bb3c37f33b8f999f6d9feec9236129be8fdd047a45ccd3d" exitCode=137
Nov 03 00:40:40.097224 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:40.097201    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-c144dd"
Nov 03 00:40:40.107057 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:40.107035    5012 scope.go:122] "RemoveContainer" containerID="818fd94c77d6023e3127ed72fcc9fad31a132ff443bcdc7c949c2fb10f5508d7"
Nov 03 00:40:40.107348 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:40.107327    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"818fd94c77d6023e3127ed72fcc9fad31a132ff443bcdc7c949c2fb10f5508d7\": container with ID starting with 818fd94c77d6023e3127ed72fcc9fad31a132ff443bcdc7c949c2fb10f5508d7 not found: ID does not exist" containerID="818fd94c77d6023e3127ed72fcc9fad31a132ff443bcdc7c949c2fb10f5508d7"
Nov 03 00:40:40.107441 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:40.107352    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"818fd94c77d6023e3127ed72fcc9fad31a132ff443bcdc7c949c2fb10f5508d7"} err="failed to get container status \"818fd94c77d6023e3127ed72fcc9fad31a132ff443bcdc7c949c2fb10f5508d7\": rpc error: code = NotFound desc = could not find container \"818fd94c77d6023e3127ed72fcc9fad31a132ff443bcdc7c949c2fb10f5508d7\": container with ID starting with 818fd94c77d6023e3127ed72fcc9fad31a132ff443bcdc7c949c2fb10f5508d7 not found: ID does not exist"
Nov 03 00:40:40.107441 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:40.107369    5012 scope.go:122] "RemoveContainer" containerID="e404d45209e015374bb3c37f33b8f999f6d9feec9236129be8fdd047a45ccd3d"
Nov 03 00:40:40.118525 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:40.118507    5012 scope.go:122] "RemoveContainer" containerID="e404d45209e015374bb3c37f33b8f999f6d9feec9236129be8fdd047a45ccd3d"
Nov 03 00:40:40.118826 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:40.118808    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e404d45209e015374bb3c37f33b8f999f6d9feec9236129be8fdd047a45ccd3d\": container with ID starting with e404d45209e015374bb3c37f33b8f999f6d9feec9236129be8fdd047a45ccd3d not found: ID does not exist" containerID="e404d45209e015374bb3c37f33b8f999f6d9feec9236129be8fdd047a45ccd3d"
Nov 03 00:40:40.118889 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:40.118833    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e404d45209e015374bb3c37f33b8f999f6d9feec9236129be8fdd047a45ccd3d"} err="failed to get container status \"e404d45209e015374bb3c37f33b8f999f6d9feec9236129be8fdd047a45ccd3d\": rpc error: code = NotFound desc = could not find container \"e404d45209e015374bb3c37f33b8f999f6d9feec9236129be8fdd047a45ccd3d\": container with ID starting with e404d45209e015374bb3c37f33b8f999f6d9feec9236129be8fdd047a45ccd3d not found: ID does not exist"
Nov 03 00:40:40.144265 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:40.144238    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"]
Nov 03 00:40:40.148241 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:40.148221    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-065de0/auth-test-model-065de0-kserve-5bbfcf4d45-ftf5g"]
Nov 03 00:40:40.153714 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:40.153691    5012 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7612-c144dd\" is forbidden: User \"system:node:ip-10-0-133-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-161.ec2.internal' and this object" podUID="e844bf55-5492-47c9-95ea-4eda5c542f34" pod="opendatahub/test-curl-7612-c144dd"
Nov 03 00:40:41.107557 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:41.107526    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7612-cca854" podUID="7b08c7f4-0fdc-4e18-a49d-b3ad1c79adea" containerName="test-curl-7612-cca854" containerID="cri-o://27ed6bd714b681610a85259ab4e2c1fe3e87e7d0589cc22e62c93fb1669e1cd5" gracePeriod=2
Nov 03 00:40:41.119143 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:41.119112    5012 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7612-cca854\" is forbidden: User \"system:node:ip-10-0-133-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-161.ec2.internal' and this object" podUID="7b08c7f4-0fdc-4e18-a49d-b3ad1c79adea" pod="opendatahub/test-curl-7612-cca854"
Nov 03 00:40:41.130788 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:41.130762    5012 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7612-c144dd\" is forbidden: User \"system:node:ip-10-0-133-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-161.ec2.internal' and this object" podUID="e844bf55-5492-47c9-95ea-4eda5c542f34" pod="opendatahub/test-curl-7612-c144dd"
Nov 03 00:40:41.246520 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:41.246482    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc53fb3c-4315-4804-b73e-c4c44c9fbca7" path="/var/lib/kubelet/pods/bc53fb3c-4315-4804-b73e-c4c44c9fbca7/volumes"
Nov 03 00:40:41.247002 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:41.246984    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e844bf55-5492-47c9-95ea-4eda5c542f34" path="/var/lib/kubelet/pods/e844bf55-5492-47c9-95ea-4eda5c542f34/volumes"
Nov 03 00:40:41.306541 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:41.306519    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-347ab8"
Nov 03 00:40:41.357048 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:41.357024    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b802e61b-eb07-4b2e-9306-18fdeb20e538-kube-api-access-j587r\" (UniqueName: \"kubernetes.io/projected/b802e61b-eb07-4b2e-9306-18fdeb20e538-kube-api-access-j587r\") pod \"b802e61b-eb07-4b2e-9306-18fdeb20e538\" (UID: \"b802e61b-eb07-4b2e-9306-18fdeb20e538\") "
Nov 03 00:40:41.359154 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:41.359127    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b802e61b-eb07-4b2e-9306-18fdeb20e538-kube-api-access-j587r" pod "b802e61b-eb07-4b2e-9306-18fdeb20e538" (UID: "b802e61b-eb07-4b2e-9306-18fdeb20e538"). InnerVolumeSpecName "kube-api-access-j587r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:41.459252 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:41.459219    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j587r\" (UniqueName: \"kubernetes.io/projected/b802e61b-eb07-4b2e-9306-18fdeb20e538-kube-api-access-j587r\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:41.917707 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:41.917668    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"]
Nov 03 00:40:41.918773 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:41.918751    5012 cpu_manager.go:414] "containerMap: removing container" podUID="bc53fb3c-4315-4804-b73e-c4c44c9fbca7" containerName="main"
Nov 03 00:40:41.918830 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:41.918778    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc53fb3c-4315-4804-b73e-c4c44c9fbca7" containerName="main"
Nov 03 00:40:41.918873 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:41.918854    5012 cpu_manager.go:414] "containerMap: removing container" podUID="7b08c7f4-0fdc-4e18-a49d-b3ad1c79adea" containerName="test-curl-7612-cca854"
Nov 03 00:40:41.918873 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:41.918863    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b08c7f4-0fdc-4e18-a49d-b3ad1c79adea" containerName="test-curl-7612-cca854"
Nov 03 00:40:41.919070 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:41.919056    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b08c7f4-0fdc-4e18-a49d-b3ad1c79adea" containerName="test-curl-7612-cca854"
Nov 03 00:40:41.919109 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:41.919083    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc53fb3c-4315-4804-b73e-c4c44c9fbca7" containerName="main"
Nov 03 00:40:41.923085 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:41.923056    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:41.945205 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:41.945176    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"]
Nov 03 00:40:41.946889 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:41.946867    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-d9e067-openshift-default-dockercfg-cx8jd\""
Nov 03 00:40:42.088935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.076475    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-workload-certs\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.089472 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.089419    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-workload-socket\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.089866 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.089836    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-token\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.090195 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.090121    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-credential-socket\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.090377 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.090355    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-data\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.090565 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.090518    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lx5l\" (UniqueName: \"kubernetes.io/projected/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-kube-api-access-6lx5l\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.094053 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.093980    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-envoy\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.103939 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.098177    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-podinfo\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.107933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.106037    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istiod-ca-cert\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.214254 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.207575    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-workload-socket\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.214254 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.207727    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-token\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.214254 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.207817    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-credential-socket\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.214254 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.207859    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-data\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.214254 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.207885    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lx5l\" (UniqueName: \"kubernetes.io/projected/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-kube-api-access-6lx5l\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.214254 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.207946    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-envoy\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.214254 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.208087    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-podinfo\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.214254 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.208137    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istiod-ca-cert\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.215713 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.214736    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-workload-certs\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.215713 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.215238    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-workload-certs\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.215713 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.215579    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-workload-socket\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.217291 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.216523    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-credential-socket\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.217291 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.216779    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-data\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.218769 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.218704    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istiod-ca-cert\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.236584 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.235623    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-podinfo\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.237510 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.237477    5012 generic.go:356] "Generic (PLEG): container finished" podID="7d9034f1-0b7c-4473-93be-4d859fb850a0" containerID="d13823617dc62bba559232a671af1bcb28e9af1518397d542be1a2d9e57342a3" exitCode=0
Nov 03 00:40:42.237645 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.237610    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p" event={"ID":"7d9034f1-0b7c-4473-93be-4d859fb850a0","Type":"ContainerDied","Data":"d13823617dc62bba559232a671af1bcb28e9af1518397d542be1a2d9e57342a3"}
Nov 03 00:40:42.240231 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.240191    5012 generic.go:356] "Generic (PLEG): container finished" podID="e9016037-dc28-41df-a3ad-5672f6f5f0c0" containerID="da45fb8997b49d1b9550527be582e6b1ff58c732a4519053c19bd0badba2fd6a" exitCode=0
Nov 03 00:40:42.240744 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.240338    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx" event={"ID":"e9016037-dc28-41df-a3ad-5672f6f5f0c0","Type":"ContainerDied","Data":"da45fb8997b49d1b9550527be582e6b1ff58c732a4519053c19bd0badba2fd6a"}
Nov 03 00:40:42.247673 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.247647    5012 generic.go:356] "Generic (PLEG): container finished" podID="2431e1a4-b1df-4516-a2bf-680be875a7d9" containerID="f01966f2449ee2cf6432e3e2285e05ba68cea6c1765bdd5a4b54236337e0563c" exitCode=0
Nov 03 00:40:42.248016 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.247992    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx" event={"ID":"2431e1a4-b1df-4516-a2bf-680be875a7d9","Type":"ContainerDied","Data":"f01966f2449ee2cf6432e3e2285e05ba68cea6c1765bdd5a4b54236337e0563c"}
Nov 03 00:40:42.250159 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.250134    5012 generic.go:356] "Generic (PLEG): container finished" podID="b802e61b-eb07-4b2e-9306-18fdeb20e538" containerID="7e01a2d4a0b53aeaf521f15879e520fba382bae438cccd78dab812693c5d6904" exitCode=137
Nov 03 00:40:42.250423 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.250407    5012 scope.go:122] "RemoveContainer" containerID="7e01a2d4a0b53aeaf521f15879e520fba382bae438cccd78dab812693c5d6904"
Nov 03 00:40:42.250825 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.250808    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-347ab8"
Nov 03 00:40:42.264858 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.264800    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lx5l\" (UniqueName: \"kubernetes.io/projected/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-kube-api-access-6lx5l\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.271405 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.271375    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-token\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.271626 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.271601    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-envoy\") pod \"e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") " pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.298987 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.298846    5012 scope.go:122] "RemoveContainer" containerID="7e01a2d4a0b53aeaf521f15879e520fba382bae438cccd78dab812693c5d6904"
Nov 03 00:40:42.299415 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:42.299377    5012 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7612-347ab8\" is forbidden: User \"system:node:ip-10-0-133-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-161.ec2.internal' and this object" podUID="b802e61b-eb07-4b2e-9306-18fdeb20e538" pod="opendatahub/test-curl-7612-347ab8"
Nov 03 00:40:42.300726 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:42.300702    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e01a2d4a0b53aeaf521f15879e520fba382bae438cccd78dab812693c5d6904\": container with ID starting with 7e01a2d4a0b53aeaf521f15879e520fba382bae438cccd78dab812693c5d6904 not found: ID does not exist" containerID="7e01a2d4a0b53aeaf521f15879e520fba382bae438cccd78dab812693c5d6904"
Nov 03 00:40:42.300820 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.300735    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e01a2d4a0b53aeaf521f15879e520fba382bae438cccd78dab812693c5d6904"} err="failed to get container status \"7e01a2d4a0b53aeaf521f15879e520fba382bae438cccd78dab812693c5d6904\": rpc error: code = NotFound desc = could not find container \"7e01a2d4a0b53aeaf521f15879e520fba382bae438cccd78dab812693c5d6904\": container with ID starting with 7e01a2d4a0b53aeaf521f15879e520fba382bae438cccd78dab812693c5d6904 not found: ID does not exist"
Nov 03 00:40:42.548740 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.548204    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:42.561507 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.561485    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:40:42.598163 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.598130    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:40:42.602441 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.602365    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:40:42.607876 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.607852    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:40:42.613729 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:42.613702    5012 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7612-347ab8\" is forbidden: User \"system:node:ip-10-0-133-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-161.ec2.internal' and this object" podUID="b802e61b-eb07-4b2e-9306-18fdeb20e538" pod="opendatahub/test-curl-7612-347ab8"
Nov 03 00:40:42.729224 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.729143    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d9034f1-0b7c-4473-93be-4d859fb850a0-kube-api-access-sdvn5\" (UniqueName: \"kubernetes.io/projected/7d9034f1-0b7c-4473-93be-4d859fb850a0-kube-api-access-sdvn5\") pod \"7d9034f1-0b7c-4473-93be-4d859fb850a0\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") "
Nov 03 00:40:42.729224 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.729184    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istiod-ca-cert\") pod \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") "
Nov 03 00:40:42.729224 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.729207    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-envoy\") pod \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") "
Nov 03 00:40:42.729522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.729240    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-envoy\") pod \"7d9034f1-0b7c-4473-93be-4d859fb850a0\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") "
Nov 03 00:40:42.729522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.729346    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-workload-socket\") pod \"7d9034f1-0b7c-4473-93be-4d859fb850a0\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") "
Nov 03 00:40:42.729522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.729415    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-podinfo\") pod \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") "
Nov 03 00:40:42.729522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.729456    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2431e1a4-b1df-4516-a2bf-680be875a7d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2431e1a4-b1df-4516-a2bf-680be875a7d9-istiod-ca-cert\") pod \"2431e1a4-b1df-4516-a2bf-680be875a7d9\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") "
Nov 03 00:40:42.729522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.729502    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-token\" (UniqueName: \"kubernetes.io/projected/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-token\") pod \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") "
Nov 03 00:40:42.729816 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.729541    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-workload-certs\") pod \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") "
Nov 03 00:40:42.729816 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.729581    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-data\") pod \"7d9034f1-0b7c-4473-93be-4d859fb850a0\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") "
Nov 03 00:40:42.729816 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.729588    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istiod-ca-cert" pod "e42db167-69a8-4280-b3d6-3ac4c6a6a42f" (UID: "e42db167-69a8-4280-b3d6-3ac4c6a6a42f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:40:42.729816 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.729609    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-token\" (UniqueName: \"kubernetes.io/projected/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-token\") pod \"2431e1a4-b1df-4516-a2bf-680be875a7d9\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") "
Nov 03 00:40:42.730336 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.730307    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-workload-socket" pod "7d9034f1-0b7c-4473-93be-4d859fb850a0" (UID: "7d9034f1-0b7c-4473-93be-4d859fb850a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.730456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.730367    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-data\") pod \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") "
Nov 03 00:40:42.730456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.730421    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istiod-ca-cert\") pod \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") "
Nov 03 00:40:42.730590 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.730498    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-credential-socket\") pod \"7d9034f1-0b7c-4473-93be-4d859fb850a0\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") "
Nov 03 00:40:42.730590 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.730557    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-podinfo\") pod \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") "
Nov 03 00:40:42.730783 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.730760    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2431e1a4-b1df-4516-a2bf-680be875a7d9-istiod-ca-cert" pod "2431e1a4-b1df-4516-a2bf-680be875a7d9" (UID: "2431e1a4-b1df-4516-a2bf-680be875a7d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:40:42.731086 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.731063    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-credential-socket\") pod \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") "
Nov 03 00:40:42.731174 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.731109    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-credential-socket\") pod \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") "
Nov 03 00:40:42.731174 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.731145    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-workload-socket\") pod \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") "
Nov 03 00:40:42.731298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.731184    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-workload-certs\") pod \"2431e1a4-b1df-4516-a2bf-680be875a7d9\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") "
Nov 03 00:40:42.731298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.731226    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-workload-certs\") pod \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") "
Nov 03 00:40:42.731420 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.731302    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-kube-api-access-2s9mh\" (UniqueName: \"kubernetes.io/projected/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-kube-api-access-2s9mh\") pod \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") "
Nov 03 00:40:42.731420 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.731335    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-workload-socket\") pod \"2431e1a4-b1df-4516-a2bf-680be875a7d9\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") "
Nov 03 00:40:42.731420 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.731383    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2431e1a4-b1df-4516-a2bf-680be875a7d9-kube-api-access-s6xrk\" (UniqueName: \"kubernetes.io/projected/2431e1a4-b1df-4516-a2bf-680be875a7d9-kube-api-access-s6xrk\") pod \"2431e1a4-b1df-4516-a2bf-680be875a7d9\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") "
Nov 03 00:40:42.731633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.731434    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-envoy\") pod \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") "
Nov 03 00:40:42.731633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.731478    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-credential-socket\") pod \"2431e1a4-b1df-4516-a2bf-680be875a7d9\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") "
Nov 03 00:40:42.731633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.731559    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-workload-socket\") pod \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") "
Nov 03 00:40:42.731633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.731595    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-token\" (UniqueName: \"kubernetes.io/projected/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-token\") pod \"7d9034f1-0b7c-4473-93be-4d859fb850a0\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") "
Nov 03 00:40:42.731848 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.731636    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-data\") pod \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\" (UID: \"e42db167-69a8-4280-b3d6-3ac4c6a6a42f\") "
Nov 03 00:40:42.731848 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.731681    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-token\" (UniqueName: \"kubernetes.io/projected/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-token\") pod \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") "
Nov 03 00:40:42.731848 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.731779    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-envoy\") pod \"2431e1a4-b1df-4516-a2bf-680be875a7d9\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") "
Nov 03 00:40:42.731848 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.731814    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-podinfo\") pod \"7d9034f1-0b7c-4473-93be-4d859fb850a0\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") "
Nov 03 00:40:42.732110 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.731848    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-workload-certs\") pod \"7d9034f1-0b7c-4473-93be-4d859fb850a0\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") "
Nov 03 00:40:42.732110 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.731882    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-podinfo\") pod \"2431e1a4-b1df-4516-a2bf-680be875a7d9\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") "
Nov 03 00:40:42.732110 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.731962    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d9034f1-0b7c-4473-93be-4d859fb850a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d9034f1-0b7c-4473-93be-4d859fb850a0-istiod-ca-cert\") pod \"7d9034f1-0b7c-4473-93be-4d859fb850a0\" (UID: \"7d9034f1-0b7c-4473-93be-4d859fb850a0\") "
Nov 03 00:40:42.732110 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.731990    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9016037-dc28-41df-a3ad-5672f6f5f0c0-kube-api-access-42cd2\" (UniqueName: \"kubernetes.io/projected/e9016037-dc28-41df-a3ad-5672f6f5f0c0-kube-api-access-42cd2\") pod \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\" (UID: \"e9016037-dc28-41df-a3ad-5672f6f5f0c0\") "
Nov 03 00:40:42.732110 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.732010    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-data\") pod \"2431e1a4-b1df-4516-a2bf-680be875a7d9\" (UID: \"2431e1a4-b1df-4516-a2bf-680be875a7d9\") "
Nov 03 00:40:42.732110 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.732030    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-workload-certs" pod "e9016037-dc28-41df-a3ad-5672f6f5f0c0" (UID: "e9016037-dc28-41df-a3ad-5672f6f5f0c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.732410 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.732150    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-data" pod "7d9034f1-0b7c-4473-93be-4d859fb850a0" (UID: "7d9034f1-0b7c-4473-93be-4d859fb850a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.732410 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.732245    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-credential-socket" pod "e42db167-69a8-4280-b3d6-3ac4c6a6a42f" (UID: "e42db167-69a8-4280-b3d6-3ac4c6a6a42f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.732410 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.732361    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-workload-socket" pod "e42db167-69a8-4280-b3d6-3ac4c6a6a42f" (UID: "e42db167-69a8-4280-b3d6-3ac4c6a6a42f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.732569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.732408    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-workload-certs" pod "e42db167-69a8-4280-b3d6-3ac4c6a6a42f" (UID: "e42db167-69a8-4280-b3d6-3ac4c6a6a42f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.732569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.732543    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-workload-certs" pod "2431e1a4-b1df-4516-a2bf-680be875a7d9" (UID: "2431e1a4-b1df-4516-a2bf-680be875a7d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.732668 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.732633    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istiod-ca-cert" pod "e9016037-dc28-41df-a3ad-5672f6f5f0c0" (UID: "e9016037-dc28-41df-a3ad-5672f6f5f0c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:40:42.732746 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.732725    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-credential-socket" pod "e9016037-dc28-41df-a3ad-5672f6f5f0c0" (UID: "e9016037-dc28-41df-a3ad-5672f6f5f0c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.732850 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.732816    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-data" pod "e42db167-69a8-4280-b3d6-3ac4c6a6a42f" (UID: "e42db167-69a8-4280-b3d6-3ac4c6a6a42f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.732953 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.732855    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-data" pod "e9016037-dc28-41df-a3ad-5672f6f5f0c0" (UID: "e9016037-dc28-41df-a3ad-5672f6f5f0c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.733033 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.732990    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.733033 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.733011    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.733033 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.733027    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2431e1a4-b1df-4516-a2bf-680be875a7d9-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.733205 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.733036    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-workload-socket" pod "e9016037-dc28-41df-a3ad-5672f6f5f0c0" (UID: "e9016037-dc28-41df-a3ad-5672f6f5f0c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.733205 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.733043    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.733205 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.733076    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.733205 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.733092    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.733205 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.733106    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.733205 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.733119    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.733205 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.733132    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.733205 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.733149    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.733205 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.733162    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.733205 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.733175    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.733205 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.733190    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.734535 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.733263    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-credential-socket" pod "7d9034f1-0b7c-4473-93be-4d859fb850a0" (UID: "7d9034f1-0b7c-4473-93be-4d859fb850a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.734535 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.733446    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-workload-socket" pod "2431e1a4-b1df-4516-a2bf-680be875a7d9" (UID: "2431e1a4-b1df-4516-a2bf-680be875a7d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.734729 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.734093    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-credential-socket" pod "2431e1a4-b1df-4516-a2bf-680be875a7d9" (UID: "2431e1a4-b1df-4516-a2bf-680be875a7d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.734729 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.734145    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-podinfo" pod "e9016037-dc28-41df-a3ad-5672f6f5f0c0" (UID: "e9016037-dc28-41df-a3ad-5672f6f5f0c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:40:42.734729 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.734608    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-data" pod "2431e1a4-b1df-4516-a2bf-680be875a7d9" (UID: "2431e1a4-b1df-4516-a2bf-680be875a7d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.734729 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.734650    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-workload-certs" pod "7d9034f1-0b7c-4473-93be-4d859fb850a0" (UID: "7d9034f1-0b7c-4473-93be-4d859fb850a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.735322 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.735180    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-envoy" pod "e9016037-dc28-41df-a3ad-5672f6f5f0c0" (UID: "e9016037-dc28-41df-a3ad-5672f6f5f0c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.735322 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.735241    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d9034f1-0b7c-4473-93be-4d859fb850a0-istiod-ca-cert" pod "7d9034f1-0b7c-4473-93be-4d859fb850a0" (UID: "7d9034f1-0b7c-4473-93be-4d859fb850a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:40:42.735322 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.735294    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-token" pod "7d9034f1-0b7c-4473-93be-4d859fb850a0" (UID: "7d9034f1-0b7c-4473-93be-4d859fb850a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:42.736497 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.736468    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9016037-dc28-41df-a3ad-5672f6f5f0c0-kube-api-access-42cd2" pod "e9016037-dc28-41df-a3ad-5672f6f5f0c0" (UID: "e9016037-dc28-41df-a3ad-5672f6f5f0c0"). InnerVolumeSpecName "kube-api-access-42cd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:42.736959 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.736923    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-kube-api-access-2s9mh" pod "e42db167-69a8-4280-b3d6-3ac4c6a6a42f" (UID: "e42db167-69a8-4280-b3d6-3ac4c6a6a42f"). InnerVolumeSpecName "kube-api-access-2s9mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:42.737610 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.737361    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2431e1a4-b1df-4516-a2bf-680be875a7d9-kube-api-access-s6xrk" pod "2431e1a4-b1df-4516-a2bf-680be875a7d9" (UID: "2431e1a4-b1df-4516-a2bf-680be875a7d9"). InnerVolumeSpecName "kube-api-access-s6xrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:42.738617 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.738584    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-podinfo" pod "2431e1a4-b1df-4516-a2bf-680be875a7d9" (UID: "2431e1a4-b1df-4516-a2bf-680be875a7d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:40:42.738767 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.738715    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-podinfo" pod "e42db167-69a8-4280-b3d6-3ac4c6a6a42f" (UID: "e42db167-69a8-4280-b3d6-3ac4c6a6a42f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:40:42.738840 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.738797    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-envoy" pod "e42db167-69a8-4280-b3d6-3ac4c6a6a42f" (UID: "e42db167-69a8-4280-b3d6-3ac4c6a6a42f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.739010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.738853    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-envoy" pod "7d9034f1-0b7c-4473-93be-4d859fb850a0" (UID: "7d9034f1-0b7c-4473-93be-4d859fb850a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.739010 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.738980    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-token" pod "e42db167-69a8-4280-b3d6-3ac4c6a6a42f" (UID: "e42db167-69a8-4280-b3d6-3ac4c6a6a42f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:42.739713 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.739400    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-token" pod "e9016037-dc28-41df-a3ad-5672f6f5f0c0" (UID: "e9016037-dc28-41df-a3ad-5672f6f5f0c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:42.740214 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.740181    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-envoy" pod "2431e1a4-b1df-4516-a2bf-680be875a7d9" (UID: "2431e1a4-b1df-4516-a2bf-680be875a7d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.740887 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.740862    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-token" pod "2431e1a4-b1df-4516-a2bf-680be875a7d9" (UID: "2431e1a4-b1df-4516-a2bf-680be875a7d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:42.743387 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.743226    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-podinfo" pod "7d9034f1-0b7c-4473-93be-4d859fb850a0" (UID: "7d9034f1-0b7c-4473-93be-4d859fb850a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:40:42.744513 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.744481    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9034f1-0b7c-4473-93be-4d859fb850a0-kube-api-access-sdvn5" pod "7d9034f1-0b7c-4473-93be-4d859fb850a0" (UID: "7d9034f1-0b7c-4473-93be-4d859fb850a0"). InnerVolumeSpecName "kube-api-access-sdvn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:42.751709 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:42.751674    5012 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7612-347ab8\" is forbidden: User \"system:node:ip-10-0-133-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-161.ec2.internal' and this object" podUID="b802e61b-eb07-4b2e-9306-18fdeb20e538" pod="opendatahub/test-curl-7612-347ab8"
Nov 03 00:40:42.835860 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.835833    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.835860 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.835862    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.836079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.835877    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.836079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.835890    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.836079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.835902    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d9034f1-0b7c-4473-93be-4d859fb850a0-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.836079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.835930    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42cd2\" (UniqueName: \"kubernetes.io/projected/e9016037-dc28-41df-a3ad-5672f6f5f0c0-kube-api-access-42cd2\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.836079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.835942    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.836079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.835956    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdvn5\" (UniqueName: \"kubernetes.io/projected/7d9034f1-0b7c-4473-93be-4d859fb850a0-kube-api-access-sdvn5\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.836079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.835967    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.836079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.835980    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.836079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.835992    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.836079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.836003    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.836079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.836015    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2431e1a4-b1df-4516-a2bf-680be875a7d9-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.836079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.836027    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9034f1-0b7c-4473-93be-4d859fb850a0-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.836079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.836040    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.836079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.836054    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s9mh\" (UniqueName: \"kubernetes.io/projected/e42db167-69a8-4280-b3d6-3ac4c6a6a42f-kube-api-access-2s9mh\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.836079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.836069    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.836079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.836082    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6xrk\" (UniqueName: \"kubernetes.io/projected/2431e1a4-b1df-4516-a2bf-680be875a7d9-kube-api-access-s6xrk\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.836951 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.836095    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.836951 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.836108    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2431e1a4-b1df-4516-a2bf-680be875a7d9-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.836951 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.836122    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9016037-dc28-41df-a3ad-5672f6f5f0c0-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.836951 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.836136    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d9034f1-0b7c-4473-93be-4d859fb850a0-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.836951 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.836148    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9016037-dc28-41df-a3ad-5672f6f5f0c0-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.883902 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:40:42.883875    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eaa89b0_054c_4c59_a4af_02eefaa98bd6.slice/crio-9680b9ac0de74fdccb474d11ea6642a3db2aa91bbdadac9b57f976c7fa5e3f1d WatchSource:0}: Error finding container 9680b9ac0de74fdccb474d11ea6642a3db2aa91bbdadac9b57f976c7fa5e3f1d: Status 404 returned error can't find the container with id 9680b9ac0de74fdccb474d11ea6642a3db2aa91bbdadac9b57f976c7fa5e3f1d
Nov 03 00:40:42.886210 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.886183    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"]
Nov 03 00:40:42.887138 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.887095    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:40:42.887222 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.887169    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:40:42.887222 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:42.887203    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:40:43.249464 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.249425    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b802e61b-eb07-4b2e-9306-18fdeb20e538" path="/var/lib/kubelet/pods/b802e61b-eb07-4b2e-9306-18fdeb20e538/volumes"
Nov 03 00:40:43.282618 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.282596    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"
Nov 03 00:40:43.282782 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.282636    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx" event={"ID":"2431e1a4-b1df-4516-a2bf-680be875a7d9","Type":"ContainerDied","Data":"c6ba0d424dc0eba415c96afc72ceb873cc98b23e9e8c98cf3b57ee502ce61f4a"}
Nov 03 00:40:43.282782 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.282714    5012 scope.go:122] "RemoveContainer" containerID="f01966f2449ee2cf6432e3e2285e05ba68cea6c1765bdd5a4b54236337e0563c"
Nov 03 00:40:43.287584 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.287554    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc" event={"ID":"5eaa89b0-054c-4c59-a4af-02eefaa98bd6","Type":"ContainerStarted","Data":"374383517c1fa174e486e662af1576622e10d8b832772519d0b6a999c47de62a"}
Nov 03 00:40:43.287696 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.287607    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc" event={"ID":"5eaa89b0-054c-4c59-a4af-02eefaa98bd6","Type":"ContainerStarted","Data":"9680b9ac0de74fdccb474d11ea6642a3db2aa91bbdadac9b57f976c7fa5e3f1d"}
Nov 03 00:40:43.289068 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.289048    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"
Nov 03 00:40:43.289068 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.289059    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p" event={"ID":"7d9034f1-0b7c-4473-93be-4d859fb850a0","Type":"ContainerDied","Data":"5706db1562b487c853ccbf8076ea911fc0a482025819ea115a1b606dbac5f580"}
Nov 03 00:40:43.290779 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.290724    5012 generic.go:356] "Generic (PLEG): container finished" podID="e42db167-69a8-4280-b3d6-3ac4c6a6a42f" containerID="a3d5ebfb03f5055a0b57d6dce75f69ceb3cbe3464d4bd2eb8f781415028f0149" exitCode=0
Nov 03 00:40:43.290952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.290818    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8" event={"ID":"e42db167-69a8-4280-b3d6-3ac4c6a6a42f","Type":"ContainerDied","Data":"a3d5ebfb03f5055a0b57d6dce75f69ceb3cbe3464d4bd2eb8f781415028f0149"}
Nov 03 00:40:43.290952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.290857    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8" event={"ID":"e42db167-69a8-4280-b3d6-3ac4c6a6a42f","Type":"ContainerDied","Data":"f94d36ba05baf57229e7bbf2991144199b1ea130570c917aa598fdd928a20e04"}
Nov 03 00:40:43.291086 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.291035    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"
Nov 03 00:40:43.293068 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.293048    5012 generic.go:356] "Generic (PLEG): container finished" podID="7b08c7f4-0fdc-4e18-a49d-b3ad1c79adea" containerID="27ed6bd714b681610a85259ab4e2c1fe3e87e7d0589cc22e62c93fb1669e1cd5" exitCode=137
Nov 03 00:40:43.294752 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.294728    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"
Nov 03 00:40:43.294752 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.294734    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx" event={"ID":"e9016037-dc28-41df-a3ad-5672f6f5f0c0","Type":"ContainerDied","Data":"a08205f5d43179e2a67897347d67f46e240fbfc8b3a3eff1a0478a2d9a46a99f"}
Nov 03 00:40:43.295951 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.295930    5012 scope.go:122] "RemoveContainer" containerID="d13823617dc62bba559232a671af1bcb28e9af1518397d542be1a2d9e57342a3"
Nov 03 00:40:43.307145 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.307125    5012 scope.go:122] "RemoveContainer" containerID="a3d5ebfb03f5055a0b57d6dce75f69ceb3cbe3464d4bd2eb8f781415028f0149"
Nov 03 00:40:43.320877 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.320855    5012 scope.go:122] "RemoveContainer" containerID="a3d5ebfb03f5055a0b57d6dce75f69ceb3cbe3464d4bd2eb8f781415028f0149"
Nov 03 00:40:43.321244 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:43.321220    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3d5ebfb03f5055a0b57d6dce75f69ceb3cbe3464d4bd2eb8f781415028f0149\": container with ID starting with a3d5ebfb03f5055a0b57d6dce75f69ceb3cbe3464d4bd2eb8f781415028f0149 not found: ID does not exist" containerID="a3d5ebfb03f5055a0b57d6dce75f69ceb3cbe3464d4bd2eb8f781415028f0149"
Nov 03 00:40:43.321333 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.321254    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3d5ebfb03f5055a0b57d6dce75f69ceb3cbe3464d4bd2eb8f781415028f0149"} err="failed to get container status \"a3d5ebfb03f5055a0b57d6dce75f69ceb3cbe3464d4bd2eb8f781415028f0149\": rpc error: code = NotFound desc = could not find container \"a3d5ebfb03f5055a0b57d6dce75f69ceb3cbe3464d4bd2eb8f781415028f0149\": container with ID starting with a3d5ebfb03f5055a0b57d6dce75f69ceb3cbe3464d4bd2eb8f781415028f0149 not found: ID does not exist"
Nov 03 00:40:43.321333 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.321278    5012 scope.go:122] "RemoveContainer" containerID="da45fb8997b49d1b9550527be582e6b1ff58c732a4519053c19bd0badba2fd6a"
Nov 03 00:40:43.335206 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.335182    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-cca854"
Nov 03 00:40:43.349587 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.349561    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"]
Nov 03 00:40:43.359495 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.359470    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lpbl8"]
Nov 03 00:40:43.392487 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.392465    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"]
Nov 03 00:40:43.398847 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.398825    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dwgpx"]
Nov 03 00:40:43.433711 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.433682    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"]
Nov 03 00:40:43.443377 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.443353    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-bb72p"]
Nov 03 00:40:43.445547 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.445529    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b08c7f4-0fdc-4e18-a49d-b3ad1c79adea-kube-api-access-frwjm\" (UniqueName: \"kubernetes.io/projected/7b08c7f4-0fdc-4e18-a49d-b3ad1c79adea-kube-api-access-frwjm\") pod \"7b08c7f4-0fdc-4e18-a49d-b3ad1c79adea\" (UID: \"7b08c7f4-0fdc-4e18-a49d-b3ad1c79adea\") "
Nov 03 00:40:43.447635 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.447605    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b08c7f4-0fdc-4e18-a49d-b3ad1c79adea-kube-api-access-frwjm" pod "7b08c7f4-0fdc-4e18-a49d-b3ad1c79adea" (UID: "7b08c7f4-0fdc-4e18-a49d-b3ad1c79adea"). InnerVolumeSpecName "kube-api-access-frwjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:43.475259 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.475233    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"]
Nov 03 00:40:43.484217 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.484194    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vgcfx"]
Nov 03 00:40:43.521022 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.520971    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc" podStartSLOduration=2.520956573 podStartE2EDuration="2.520956573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:43.518287245 +0000 UTC m=+2126.397871288" watchObservedRunningTime="2026-11-03 00:40:43.520956573 +0000 UTC m=+2126.400540610"
Nov 03 00:40:43.547784 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.547752    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frwjm\" (UniqueName: \"kubernetes.io/projected/7b08c7f4-0fdc-4e18-a49d-b3ad1c79adea-kube-api-access-frwjm\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:40:43.549416 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.549394    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:43.555442 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:43.555423    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:44.307043 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:44.307012    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-cca854"
Nov 03 00:40:44.307543 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:44.307011    5012 scope.go:122] "RemoveContainer" containerID="27ed6bd714b681610a85259ab4e2c1fe3e87e7d0589cc22e62c93fb1669e1cd5"
Nov 03 00:40:44.308371 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:44.308283    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:44.312130 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:44.312107    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:40:44.371957 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:40:44.371882    5012 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7612-cca854\" is forbidden: User \"system:node:ip-10-0-133-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-161.ec2.internal' and this object" podUID="7b08c7f4-0fdc-4e18-a49d-b3ad1c79adea" pod="opendatahub/test-curl-7612-cca854"
Nov 03 00:40:45.245210 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:45.245179    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2431e1a4-b1df-4516-a2bf-680be875a7d9" path="/var/lib/kubelet/pods/2431e1a4-b1df-4516-a2bf-680be875a7d9/volumes"
Nov 03 00:40:45.245556 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:45.245541    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b08c7f4-0fdc-4e18-a49d-b3ad1c79adea" path="/var/lib/kubelet/pods/7b08c7f4-0fdc-4e18-a49d-b3ad1c79adea/volumes"
Nov 03 00:40:45.246011 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:45.245995    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d9034f1-0b7c-4473-93be-4d859fb850a0" path="/var/lib/kubelet/pods/7d9034f1-0b7c-4473-93be-4d859fb850a0/volumes"
Nov 03 00:40:45.246731 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:45.246715    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e42db167-69a8-4280-b3d6-3ac4c6a6a42f" path="/var/lib/kubelet/pods/e42db167-69a8-4280-b3d6-3ac4c6a6a42f/volumes"
Nov 03 00:40:45.247185 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:45.247171    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9016037-dc28-41df-a3ad-5672f6f5f0c0" path="/var/lib/kubelet/pods/e9016037-dc28-41df-a3ad-5672f6f5f0c0/volumes"
Nov 03 00:40:47.711941 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.711896    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5"]
Nov 03 00:40:47.712551 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.712537    5012 cpu_manager.go:414] "containerMap: removing container" podUID="7d9034f1-0b7c-4473-93be-4d859fb850a0" containerName="istio-proxy"
Nov 03 00:40:47.712599 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.712551    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d9034f1-0b7c-4473-93be-4d859fb850a0" containerName="istio-proxy"
Nov 03 00:40:47.712599 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.712585    5012 cpu_manager.go:414] "containerMap: removing container" podUID="e9016037-dc28-41df-a3ad-5672f6f5f0c0" containerName="istio-proxy"
Nov 03 00:40:47.712599 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.712592    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9016037-dc28-41df-a3ad-5672f6f5f0c0" containerName="istio-proxy"
Nov 03 00:40:47.712703 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.712606    5012 cpu_manager.go:414] "containerMap: removing container" podUID="e42db167-69a8-4280-b3d6-3ac4c6a6a42f" containerName="istio-proxy"
Nov 03 00:40:47.712703 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.712611    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e42db167-69a8-4280-b3d6-3ac4c6a6a42f" containerName="istio-proxy"
Nov 03 00:40:47.712703 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.712623    5012 cpu_manager.go:414] "containerMap: removing container" podUID="2431e1a4-b1df-4516-a2bf-680be875a7d9" containerName="istio-proxy"
Nov 03 00:40:47.712703 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.712629    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2431e1a4-b1df-4516-a2bf-680be875a7d9" containerName="istio-proxy"
Nov 03 00:40:47.712829 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.712724    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="e42db167-69a8-4280-b3d6-3ac4c6a6a42f" containerName="istio-proxy"
Nov 03 00:40:47.712829 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.712734    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="2431e1a4-b1df-4516-a2bf-680be875a7d9" containerName="istio-proxy"
Nov 03 00:40:47.712829 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.712741    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d9034f1-0b7c-4473-93be-4d859fb850a0" containerName="istio-proxy"
Nov 03 00:40:47.712829 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.712751    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9016037-dc28-41df-a3ad-5672f6f5f0c0" containerName="istio-proxy"
Nov 03 00:40:47.717776 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.717757    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5"
Nov 03 00:40:47.730633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.730612    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-d9e067\""
Nov 03 00:40:47.747815 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.747788    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5"]
Nov 03 00:40:47.779554 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.779523    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl"]
Nov 03 00:40:47.784792 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.784771    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl"
Nov 03 00:40:47.798600 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.798578    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-d9e067-dockercfg-tdlhj\""
Nov 03 00:40:47.801636 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.801612    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl"]
Nov 03 00:40:47.803374 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.803351    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w8ml\" (UniqueName: \"kubernetes.io/projected/cafedf4f-719f-4e62-9f27-28f2e2017131-kube-api-access-5w8ml\") pod \"payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5\" (UID: \"cafedf4f-719f-4e62-9f27-28f2e2017131\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5"
Nov 03 00:40:47.803478 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.803400    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmv4g\" (UniqueName: \"kubernetes.io/projected/d4aa636c-5490-4e79-9b11-3284af3e64a6-kube-api-access-mmv4g\") pod \"payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl\" (UID: \"d4aa636c-5490-4e79-9b11-3284af3e64a6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl"
Nov 03 00:40:47.803536 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.803515    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cafedf4f-719f-4e62-9f27-28f2e2017131-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5\" (UID: \"cafedf4f-719f-4e62-9f27-28f2e2017131\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5"
Nov 03 00:40:47.803590 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.803569    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4aa636c-5490-4e79-9b11-3284af3e64a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl\" (UID: \"d4aa636c-5490-4e79-9b11-3284af3e64a6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl"
Nov 03 00:40:47.942930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.940850    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w8ml\" (UniqueName: \"kubernetes.io/projected/cafedf4f-719f-4e62-9f27-28f2e2017131-kube-api-access-5w8ml\") pod \"payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5\" (UID: \"cafedf4f-719f-4e62-9f27-28f2e2017131\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5"
Nov 03 00:40:47.942930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.940955    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmv4g\" (UniqueName: \"kubernetes.io/projected/d4aa636c-5490-4e79-9b11-3284af3e64a6-kube-api-access-mmv4g\") pod \"payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl\" (UID: \"d4aa636c-5490-4e79-9b11-3284af3e64a6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl"
Nov 03 00:40:47.942930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.941050    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cafedf4f-719f-4e62-9f27-28f2e2017131-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5\" (UID: \"cafedf4f-719f-4e62-9f27-28f2e2017131\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5"
Nov 03 00:40:47.942930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.941121    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4aa636c-5490-4e79-9b11-3284af3e64a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl\" (UID: \"d4aa636c-5490-4e79-9b11-3284af3e64a6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl"
Nov 03 00:40:47.951775 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.951083    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cafedf4f-719f-4e62-9f27-28f2e2017131-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5\" (UID: \"cafedf4f-719f-4e62-9f27-28f2e2017131\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5"
Nov 03 00:40:47.955284 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:47.955254    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4aa636c-5490-4e79-9b11-3284af3e64a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl\" (UID: \"d4aa636c-5490-4e79-9b11-3284af3e64a6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl"
Nov 03 00:40:48.005271 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:48.005171    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmv4g\" (UniqueName: \"kubernetes.io/projected/d4aa636c-5490-4e79-9b11-3284af3e64a6-kube-api-access-mmv4g\") pod \"payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl\" (UID: \"d4aa636c-5490-4e79-9b11-3284af3e64a6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl"
Nov 03 00:40:48.019453 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:48.013732    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w8ml\" (UniqueName: \"kubernetes.io/projected/cafedf4f-719f-4e62-9f27-28f2e2017131-kube-api-access-5w8ml\") pod \"payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5\" (UID: \"cafedf4f-719f-4e62-9f27-28f2e2017131\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5"
Nov 03 00:40:48.032608 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:48.032574    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5"
Nov 03 00:40:48.103133 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:48.102840    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl"
Nov 03 00:40:48.358200 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:48.358176    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5"]
Nov 03 00:40:48.358395 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:40:48.358377    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcafedf4f_719f_4e62_9f27_28f2e2017131.slice/crio-d53acb0d60a769c4f0fbbf2da01933d62f4b38cd347e953bee0a24de8776e99f WatchSource:0}: Error finding container d53acb0d60a769c4f0fbbf2da01933d62f4b38cd347e953bee0a24de8776e99f: Status 404 returned error can't find the container with id d53acb0d60a769c4f0fbbf2da01933d62f4b38cd347e953bee0a24de8776e99f
Nov 03 00:40:48.416717 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:48.416691    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl"]
Nov 03 00:40:48.417326 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:40:48.417301    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4aa636c_5490_4e79_9b11_3284af3e64a6.slice/crio-38c7548ad003d582c500fe1a996f005c8d428dcdae44cc7b754506d64bf8104e WatchSource:0}: Error finding container 38c7548ad003d582c500fe1a996f005c8d428dcdae44cc7b754506d64bf8104e: Status 404 returned error can't find the container with id 38c7548ad003d582c500fe1a996f005c8d428dcdae44cc7b754506d64bf8104e
Nov 03 00:40:49.351999 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:49.351956    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5" event={"ID":"cafedf4f-719f-4e62-9f27-28f2e2017131","Type":"ContainerStarted","Data":"4c379a996cbbb27f2e418e4ce0af3d2e442b6c125e65d2f345f8eec52184d1ff"}
Nov 03 00:40:49.352459 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:49.352030    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5"
Nov 03 00:40:49.352459 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:49.352056    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5" event={"ID":"cafedf4f-719f-4e62-9f27-28f2e2017131","Type":"ContainerStarted","Data":"d53acb0d60a769c4f0fbbf2da01933d62f4b38cd347e953bee0a24de8776e99f"}
Nov 03 00:40:49.353695 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:49.353663    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl" event={"ID":"d4aa636c-5490-4e79-9b11-3284af3e64a6","Type":"ContainerStarted","Data":"88ac4c55a2000b4ce1146da1ab9a314af0e2e3d0317c7cc81358a71873f3d91d"}
Nov 03 00:40:49.353848 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:49.353709    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl" event={"ID":"d4aa636c-5490-4e79-9b11-3284af3e64a6","Type":"ContainerStarted","Data":"38c7548ad003d582c500fe1a996f005c8d428dcdae44cc7b754506d64bf8104e"}
Nov 03 00:40:49.353848 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:49.353732    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl"
Nov 03 00:40:49.387245 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:49.387204    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5" podStartSLOduration=2.38719274 podStartE2EDuration="2.38719274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:49.385263515 +0000 UTC m=+2132.264847557" watchObservedRunningTime="2026-11-03 00:40:49.38719274 +0000 UTC m=+2132.266776782"
Nov 03 00:40:49.424815 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:49.424769    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl" podStartSLOduration=2.424756932 podStartE2EDuration="2.424756932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:49.421043394 +0000 UTC m=+2132.300627436" watchObservedRunningTime="2026-11-03 00:40:49.424756932 +0000 UTC m=+2132.304340976"
Nov 03 00:40:50.669553 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:50.669521    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv"]
Nov 03 00:40:50.674192 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:50.674168    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv"
Nov 03 00:40:50.692729 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:50.692706    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv"]
Nov 03 00:40:50.698140 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:50.698100    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-099583-serving-cert\""
Nov 03 00:40:50.777027 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:50.776997    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv\" (UID: \"a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv"
Nov 03 00:40:50.777193 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:50.777032    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv\" (UID: \"a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv"
Nov 03 00:40:50.777193 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:50.777065    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv\" (UID: \"a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv"
Nov 03 00:40:50.777332 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:50.777216    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kwhd\" (UniqueName: \"kubernetes.io/projected/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-kube-api-access-5kwhd\") pod \"maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv\" (UID: \"a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv"
Nov 03 00:40:50.878184 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:50.878159    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kwhd\" (UniqueName: \"kubernetes.io/projected/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-kube-api-access-5kwhd\") pod \"maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv\" (UID: \"a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv"
Nov 03 00:40:50.878331 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:50.878293    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv\" (UID: \"a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv"
Nov 03 00:40:50.878331 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:50.878317    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv\" (UID: \"a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv"
Nov 03 00:40:50.878437 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:50.878338    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv\" (UID: \"a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv"
Nov 03 00:40:50.879063 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:50.879038    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv\" (UID: \"a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv"
Nov 03 00:40:50.880862 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:50.880839    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv\" (UID: \"a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv"
Nov 03 00:40:50.880862 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:50.880849    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv\" (UID: \"a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv"
Nov 03 00:40:50.894986 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:50.894962    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kwhd\" (UniqueName: \"kubernetes.io/projected/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-kube-api-access-5kwhd\") pod \"maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv\" (UID: \"a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv"
Nov 03 00:40:50.989730 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:50.989540    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv"
Nov 03 00:40:51.302225 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:40:51.302198    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d013a8_d1ca_4ff2_8bb9_bbe5fba4de22.slice/crio-f24711105ab1e26452d6969ee4830505d1c3a318c43fd8ce444893bf3988a78c WatchSource:0}: Error finding container f24711105ab1e26452d6969ee4830505d1c3a318c43fd8ce444893bf3988a78c: Status 404 returned error can't find the container with id f24711105ab1e26452d6969ee4830505d1c3a318c43fd8ce444893bf3988a78c
Nov 03 00:40:51.302487 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:51.302463    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv"]
Nov 03 00:40:51.380080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:51.380040    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv" event={"ID":"a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22","Type":"ContainerStarted","Data":"f24711105ab1e26452d6969ee4830505d1c3a318c43fd8ce444893bf3988a78c"}
Nov 03 00:40:52.389839 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:52.389795    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv" event={"ID":"a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22","Type":"ContainerStarted","Data":"c2f23afb85a9925d108df5879d073b731900d756e87326421707c5989498a21c"}
Nov 03 00:40:52.390318 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:52.389861    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv"
Nov 03 00:40:52.426639 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:52.426588    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv" podStartSLOduration=2.080672989 podStartE2EDuration="2.426574052s" totalImagesPullingTime="345.901063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:52.424428567 +0000 UTC m=+2135.304012614" watchObservedRunningTime="2026-11-03 00:40:52.426574052 +0000 UTC m=+2135.306158121"
Nov 03 00:40:54.479117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:54.479076    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"]
Nov 03 00:40:58.402413 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:40:58.402379    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv"
Nov 03 00:41:00.365147 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:00.365114    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5"
Nov 03 00:41:00.365598 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:00.365475    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl"
Nov 03 00:41:07.026262 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:07.026228    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"]
Nov 03 00:41:07.026757 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:07.026538    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm" podUID="f996bace-902b-4e8d-95bc-43f248835e98" containerName="istio-proxy" containerID="cri-o://edcd096742c694466ab096520881c14df0c07022639e0aa7965acc1dcccb12c4" gracePeriod=30
Nov 03 00:41:07.364628 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:41:07.364583    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:41:09.502536 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:09.502486    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq" podUID="3bff9372-676b-47d2-89e4-f423a422cf1d" containerName="main" containerID="cri-o://11076b1a0ed2174e8f11a2bda8e8f8dfb13b7133cd9cf06d33e20a624b369d69" gracePeriod=45
Nov 03 00:41:09.505136 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:09.505099    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq" podUID="3bff9372-676b-47d2-89e4-f423a422cf1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.230:8000/ready\": dial tcp 10.132.0.230:8000: connect: connection refused"
Nov 03 00:41:09.788804 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:09.788778    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:41:09.952533 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:09.952501    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-model-cache\") pod \"3bff9372-676b-47d2-89e4-f423a422cf1d\" (UID: \"3bff9372-676b-47d2-89e4-f423a422cf1d\") "
Nov 03 00:41:09.952698 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:09.952545    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bff9372-676b-47d2-89e4-f423a422cf1d-kube-api-access-dzft9\" (UniqueName: \"kubernetes.io/projected/3bff9372-676b-47d2-89e4-f423a422cf1d-kube-api-access-dzft9\") pod \"3bff9372-676b-47d2-89e4-f423a422cf1d\" (UID: \"3bff9372-676b-47d2-89e4-f423a422cf1d\") "
Nov 03 00:41:09.952749 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:09.952706    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bff9372-676b-47d2-89e4-f423a422cf1d-tls-certs\" (UniqueName: \"kubernetes.io/secret/3bff9372-676b-47d2-89e4-f423a422cf1d-tls-certs\") pod \"3bff9372-676b-47d2-89e4-f423a422cf1d\" (UID: \"3bff9372-676b-47d2-89e4-f423a422cf1d\") "
Nov 03 00:41:09.952799 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:09.952750    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-home\" (UniqueName: \"kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-home\") pod \"3bff9372-676b-47d2-89e4-f423a422cf1d\" (UID: \"3bff9372-676b-47d2-89e4-f423a422cf1d\") "
Nov 03 00:41:09.952853 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:09.952803    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-model-cache" pod "3bff9372-676b-47d2-89e4-f423a422cf1d" (UID: "3bff9372-676b-47d2-89e4-f423a422cf1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:09.952853 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:09.952827    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-tmp-dir\") pod \"3bff9372-676b-47d2-89e4-f423a422cf1d\" (UID: \"3bff9372-676b-47d2-89e4-f423a422cf1d\") "
Nov 03 00:41:09.952972 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:09.952946    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-dshm\") pod \"3bff9372-676b-47d2-89e4-f423a422cf1d\" (UID: \"3bff9372-676b-47d2-89e4-f423a422cf1d\") "
Nov 03 00:41:09.953061 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:09.953037    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-home" pod "3bff9372-676b-47d2-89e4-f423a422cf1d" (UID: "3bff9372-676b-47d2-89e4-f423a422cf1d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:09.953155 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:09.953130    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-tmp-dir" pod "3bff9372-676b-47d2-89e4-f423a422cf1d" (UID: "3bff9372-676b-47d2-89e4-f423a422cf1d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:09.953502 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:09.953485    5012 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-model-cache\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:09.953563 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:09.953505    5012 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-home\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:09.953563 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:09.953515    5012 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-tmp-dir\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:09.955033 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:09.955004    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-dshm" pod "3bff9372-676b-47d2-89e4-f423a422cf1d" (UID: "3bff9372-676b-47d2-89e4-f423a422cf1d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:09.955183 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:09.955167    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bff9372-676b-47d2-89e4-f423a422cf1d-kube-api-access-dzft9" pod "3bff9372-676b-47d2-89e4-f423a422cf1d" (UID: "3bff9372-676b-47d2-89e4-f423a422cf1d"). InnerVolumeSpecName "kube-api-access-dzft9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:09.955235 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:09.955175    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bff9372-676b-47d2-89e4-f423a422cf1d-tls-certs" pod "3bff9372-676b-47d2-89e4-f423a422cf1d" (UID: "3bff9372-676b-47d2-89e4-f423a422cf1d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:41:10.054514 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:10.054419    5012 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bff9372-676b-47d2-89e4-f423a422cf1d-tls-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:10.054514 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:10.054457    5012 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bff9372-676b-47d2-89e4-f423a422cf1d-dshm\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:10.054514 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:10.054469    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzft9\" (UniqueName: \"kubernetes.io/projected/3bff9372-676b-47d2-89e4-f423a422cf1d-kube-api-access-dzft9\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:10.605954 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:10.605900    5012 generic.go:356] "Generic (PLEG): container finished" podID="3bff9372-676b-47d2-89e4-f423a422cf1d" containerID="11076b1a0ed2174e8f11a2bda8e8f8dfb13b7133cd9cf06d33e20a624b369d69" exitCode=0
Nov 03 00:41:10.605954 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:10.605942    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq" event={"ID":"3bff9372-676b-47d2-89e4-f423a422cf1d","Type":"ContainerDied","Data":"11076b1a0ed2174e8f11a2bda8e8f8dfb13b7133cd9cf06d33e20a624b369d69"}
Nov 03 00:41:10.606685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:10.605991    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"
Nov 03 00:41:10.606685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:10.606012    5012 scope.go:122] "RemoveContainer" containerID="11076b1a0ed2174e8f11a2bda8e8f8dfb13b7133cd9cf06d33e20a624b369d69"
Nov 03 00:41:10.606685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:10.605999    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq" event={"ID":"3bff9372-676b-47d2-89e4-f423a422cf1d","Type":"ContainerDied","Data":"7b6bdedb4476b2a7c6f133554cc6afecde635a0c3f9ea268da32befa50bc537b"}
Nov 03 00:41:10.618385 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:10.618366    5012 scope.go:122] "RemoveContainer" containerID="11076b1a0ed2174e8f11a2bda8e8f8dfb13b7133cd9cf06d33e20a624b369d69"
Nov 03 00:41:10.618637 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:41:10.618620    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11076b1a0ed2174e8f11a2bda8e8f8dfb13b7133cd9cf06d33e20a624b369d69\": container with ID starting with 11076b1a0ed2174e8f11a2bda8e8f8dfb13b7133cd9cf06d33e20a624b369d69 not found: ID does not exist" containerID="11076b1a0ed2174e8f11a2bda8e8f8dfb13b7133cd9cf06d33e20a624b369d69"
Nov 03 00:41:10.618695 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:10.618645    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11076b1a0ed2174e8f11a2bda8e8f8dfb13b7133cd9cf06d33e20a624b369d69"} err="failed to get container status \"11076b1a0ed2174e8f11a2bda8e8f8dfb13b7133cd9cf06d33e20a624b369d69\": rpc error: code = NotFound desc = could not find container \"11076b1a0ed2174e8f11a2bda8e8f8dfb13b7133cd9cf06d33e20a624b369d69\": container with ID starting with 11076b1a0ed2174e8f11a2bda8e8f8dfb13b7133cd9cf06d33e20a624b369d69 not found: ID does not exist"
Nov 03 00:41:10.653515 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:10.653486    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"]
Nov 03 00:41:10.663865 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:10.663839    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-359f71/auth-test-model-359f71-kserve-859cfd64d5-p87rq"]
Nov 03 00:41:11.246396 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:11.246367    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bff9372-676b-47d2-89e4-f423a422cf1d" path="/var/lib/kubelet/pods/3bff9372-676b-47d2-89e4-f423a422cf1d/volumes"
Nov 03 00:41:11.485641 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:11.485600    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft"]
Nov 03 00:41:11.486456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:11.486421    5012 cpu_manager.go:414] "containerMap: removing container" podUID="3bff9372-676b-47d2-89e4-f423a422cf1d" containerName="main"
Nov 03 00:41:11.486456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:11.486453    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bff9372-676b-47d2-89e4-f423a422cf1d" containerName="main"
Nov 03 00:41:11.486678 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:11.486664    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bff9372-676b-47d2-89e4-f423a422cf1d" containerName="main"
Nov 03 00:41:11.491716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:11.491695    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft"
Nov 03 00:41:11.508848 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:11.508821    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft"]
Nov 03 00:41:11.514117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:11.514091    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-d9e067-serving-cert\""
Nov 03 00:41:11.572601 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:11.572563    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft\" (UID: \"5cda386d-6a1d-4fd9-95c5-1a68750e10b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft"
Nov 03 00:41:11.572768 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:11.572648    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-metrics-certs\") pod \"maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft\" (UID: \"5cda386d-6a1d-4fd9-95c5-1a68750e10b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft"
Nov 03 00:41:11.572768 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:11.572753    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft\" (UID: \"5cda386d-6a1d-4fd9-95c5-1a68750e10b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft"
Nov 03 00:41:11.572884 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:11.572867    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg9dv\" (UniqueName: \"kubernetes.io/projected/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-kube-api-access-tg9dv\") pod \"maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft\" (UID: \"5cda386d-6a1d-4fd9-95c5-1a68750e10b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft"
Nov 03 00:41:11.674420 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:11.674387    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg9dv\" (UniqueName: \"kubernetes.io/projected/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-kube-api-access-tg9dv\") pod \"maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft\" (UID: \"5cda386d-6a1d-4fd9-95c5-1a68750e10b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft"
Nov 03 00:41:11.674773 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:11.674534    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft\" (UID: \"5cda386d-6a1d-4fd9-95c5-1a68750e10b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft"
Nov 03 00:41:11.674773 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:11.674597    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-metrics-certs\") pod \"maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft\" (UID: \"5cda386d-6a1d-4fd9-95c5-1a68750e10b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft"
Nov 03 00:41:11.674773 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:11.674670    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft\" (UID: \"5cda386d-6a1d-4fd9-95c5-1a68750e10b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft"
Nov 03 00:41:11.675265 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:11.675242    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft\" (UID: \"5cda386d-6a1d-4fd9-95c5-1a68750e10b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft"
Nov 03 00:41:11.677172 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:11.677145    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft\" (UID: \"5cda386d-6a1d-4fd9-95c5-1a68750e10b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft"
Nov 03 00:41:11.677172 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:11.677160    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-metrics-certs\") pod \"maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft\" (UID: \"5cda386d-6a1d-4fd9-95c5-1a68750e10b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft"
Nov 03 00:41:11.692072 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:11.692048    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg9dv\" (UniqueName: \"kubernetes.io/projected/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-kube-api-access-tg9dv\") pod \"maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft\" (UID: \"5cda386d-6a1d-4fd9-95c5-1a68750e10b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft"
Nov 03 00:41:11.810749 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:11.810665    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft"
Nov 03 00:41:12.142019 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:41:12.141987    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cda386d_6a1d_4fd9_95c5_1a68750e10b0.slice/crio-02d6670cc77dbfe50a11e5a181a3a7a938f142631ba0eb3c1e867a52bd37ff4f WatchSource:0}: Error finding container 02d6670cc77dbfe50a11e5a181a3a7a938f142631ba0eb3c1e867a52bd37ff4f: Status 404 returned error can't find the container with id 02d6670cc77dbfe50a11e5a181a3a7a938f142631ba0eb3c1e867a52bd37ff4f
Nov 03 00:41:12.142019 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.141997    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft"]
Nov 03 00:41:12.278533 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.278503    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:41:12.390282 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.390208    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-istio-envoy\") pod \"f996bace-902b-4e8d-95bc-43f248835e98\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") "
Nov 03 00:41:12.390471 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.390323    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-credential-socket\") pod \"f996bace-902b-4e8d-95bc-43f248835e98\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") "
Nov 03 00:41:12.390471 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.390349    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f996bace-902b-4e8d-95bc-43f248835e98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f996bace-902b-4e8d-95bc-43f248835e98-istio-podinfo\") pod \"f996bace-902b-4e8d-95bc-43f248835e98\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") "
Nov 03 00:41:12.390471 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.390406    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-istio-data\") pod \"f996bace-902b-4e8d-95bc-43f248835e98\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") "
Nov 03 00:41:12.390664 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.390491    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f996bace-902b-4e8d-95bc-43f248835e98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f996bace-902b-4e8d-95bc-43f248835e98-istiod-ca-cert\") pod \"f996bace-902b-4e8d-95bc-43f248835e98\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") "
Nov 03 00:41:12.390664 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.390587    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f996bace-902b-4e8d-95bc-43f248835e98-kube-api-access-758rh\" (UniqueName: \"kubernetes.io/projected/f996bace-902b-4e8d-95bc-43f248835e98-kube-api-access-758rh\") pod \"f996bace-902b-4e8d-95bc-43f248835e98\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") "
Nov 03 00:41:12.390777 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.390684    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-credential-socket" pod "f996bace-902b-4e8d-95bc-43f248835e98" (UID: "f996bace-902b-4e8d-95bc-43f248835e98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.390833 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.390790    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-workload-certs\") pod \"f996bace-902b-4e8d-95bc-43f248835e98\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") "
Nov 03 00:41:12.390896 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.390837    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f996bace-902b-4e8d-95bc-43f248835e98-istio-token\" (UniqueName: \"kubernetes.io/projected/f996bace-902b-4e8d-95bc-43f248835e98-istio-token\") pod \"f996bace-902b-4e8d-95bc-43f248835e98\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") "
Nov 03 00:41:12.390896 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.390869    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-workload-socket\") pod \"f996bace-902b-4e8d-95bc-43f248835e98\" (UID: \"f996bace-902b-4e8d-95bc-43f248835e98\") "
Nov 03 00:41:12.391133 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.391104    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-istio-data" pod "f996bace-902b-4e8d-95bc-43f248835e98" (UID: "f996bace-902b-4e8d-95bc-43f248835e98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.391349 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.391324    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-workload-certs" pod "f996bace-902b-4e8d-95bc-43f248835e98" (UID: "f996bace-902b-4e8d-95bc-43f248835e98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.391439 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.391345    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-workload-socket" pod "f996bace-902b-4e8d-95bc-43f248835e98" (UID: "f996bace-902b-4e8d-95bc-43f248835e98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.391439 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.391357    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f996bace-902b-4e8d-95bc-43f248835e98-istiod-ca-cert" pod "f996bace-902b-4e8d-95bc-43f248835e98" (UID: "f996bace-902b-4e8d-95bc-43f248835e98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:41:12.391792 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.391769    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.391871 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.391799    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.391871 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.391814    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f996bace-902b-4e8d-95bc-43f248835e98-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.391871 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.391827    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.391871 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.391839    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.393137 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.393108    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-istio-envoy" pod "f996bace-902b-4e8d-95bc-43f248835e98" (UID: "f996bace-902b-4e8d-95bc-43f248835e98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.393260 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.393241    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f996bace-902b-4e8d-95bc-43f248835e98-kube-api-access-758rh" pod "f996bace-902b-4e8d-95bc-43f248835e98" (UID: "f996bace-902b-4e8d-95bc-43f248835e98"). InnerVolumeSpecName "kube-api-access-758rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:12.393553 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.393527    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f996bace-902b-4e8d-95bc-43f248835e98-istio-token" pod "f996bace-902b-4e8d-95bc-43f248835e98" (UID: "f996bace-902b-4e8d-95bc-43f248835e98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:12.394365 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.394344    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f996bace-902b-4e8d-95bc-43f248835e98-istio-podinfo" pod "f996bace-902b-4e8d-95bc-43f248835e98" (UID: "f996bace-902b-4e8d-95bc-43f248835e98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:41:12.493333 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.493302    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f996bace-902b-4e8d-95bc-43f248835e98-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.493449 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.493340    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-758rh\" (UniqueName: \"kubernetes.io/projected/f996bace-902b-4e8d-95bc-43f248835e98-kube-api-access-758rh\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.493449 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.493359    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f996bace-902b-4e8d-95bc-43f248835e98-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.493449 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.493372    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f996bace-902b-4e8d-95bc-43f248835e98-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.627508 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.627468    5012 generic.go:356] "Generic (PLEG): container finished" podID="f996bace-902b-4e8d-95bc-43f248835e98" containerID="edcd096742c694466ab096520881c14df0c07022639e0aa7965acc1dcccb12c4" exitCode=0
Nov 03 00:41:12.627685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.627538    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"
Nov 03 00:41:12.627685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.627549    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm" event={"ID":"f996bace-902b-4e8d-95bc-43f248835e98","Type":"ContainerDied","Data":"edcd096742c694466ab096520881c14df0c07022639e0aa7965acc1dcccb12c4"}
Nov 03 00:41:12.627685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.627607    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm" event={"ID":"f996bace-902b-4e8d-95bc-43f248835e98","Type":"ContainerDied","Data":"955c8d6a3541a4d8a5fafba30b2bea42c35c714f4953ec216c6f56c0465681c3"}
Nov 03 00:41:12.627685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.627637    5012 scope.go:122] "RemoveContainer" containerID="edcd096742c694466ab096520881c14df0c07022639e0aa7965acc1dcccb12c4"
Nov 03 00:41:12.629862 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.629839    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft" event={"ID":"5cda386d-6a1d-4fd9-95c5-1a68750e10b0","Type":"ContainerStarted","Data":"d27aebf916c104eac2ffa546ede69c2ba0ae12b95fc4308346c4e4db5bca6653"}
Nov 03 00:41:12.630142 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.629886    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft" event={"ID":"5cda386d-6a1d-4fd9-95c5-1a68750e10b0","Type":"ContainerStarted","Data":"02d6670cc77dbfe50a11e5a181a3a7a938f142631ba0eb3c1e867a52bd37ff4f"}
Nov 03 00:41:12.630142 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.629955    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft"
Nov 03 00:41:12.641027 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.640998    5012 scope.go:122] "RemoveContainer" containerID="edcd096742c694466ab096520881c14df0c07022639e0aa7965acc1dcccb12c4"
Nov 03 00:41:12.641271 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:41:12.641252    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edcd096742c694466ab096520881c14df0c07022639e0aa7965acc1dcccb12c4\": container with ID starting with edcd096742c694466ab096520881c14df0c07022639e0aa7965acc1dcccb12c4 not found: ID does not exist" containerID="edcd096742c694466ab096520881c14df0c07022639e0aa7965acc1dcccb12c4"
Nov 03 00:41:12.641324 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.641281    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edcd096742c694466ab096520881c14df0c07022639e0aa7965acc1dcccb12c4"} err="failed to get container status \"edcd096742c694466ab096520881c14df0c07022639e0aa7965acc1dcccb12c4\": rpc error: code = NotFound desc = could not find container \"edcd096742c694466ab096520881c14df0c07022639e0aa7965acc1dcccb12c4\": container with ID starting with edcd096742c694466ab096520881c14df0c07022639e0aa7965acc1dcccb12c4 not found: ID does not exist"
Nov 03 00:41:12.676780 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.676725    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft" podStartSLOduration=1.405853293 podStartE2EDuration="1.676712173s" totalImagesPullingTime="270.85888ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:12.67577909 +0000 UTC m=+2155.555363136" watchObservedRunningTime="2026-11-03 00:41:12.676712173 +0000 UTC m=+2155.556296214"
Nov 03 00:41:12.711427 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.711399    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"]
Nov 03 00:41:12.717047 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:12.717018    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-djbxm"]
Nov 03 00:41:13.245133 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:13.245101    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f996bace-902b-4e8d-95bc-43f248835e98" path="/var/lib/kubelet/pods/f996bace-902b-4e8d-95bc-43f248835e98/volumes"
Nov 03 00:41:15.077188 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:15.077154    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"]
Nov 03 00:41:18.662234 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:18.662198    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft"
Nov 03 00:41:20.036278 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:20.036248    5012 scope.go:122] "RemoveContainer" containerID="ac62147ea9623a20f7827cde25fe97c2d5ff50c336f66299047aa7d3ed46c43d"
Nov 03 00:41:20.047038 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:20.047002    5012 scope.go:122] "RemoveContainer" containerID="40b53c616e3fb04c117942fdab8af7b39de3e18a9618fdd07f670506636f7834"
Nov 03 00:41:20.056488 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:20.056468    5012 scope.go:122] "RemoveContainer" containerID="fe825e977a05c3ce2c668b568e3e156f97817eed629d58d9e73ad34a7ab39d86"
Nov 03 00:41:22.117773 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:22.117737    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"]
Nov 03 00:41:22.118240 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:22.117982    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6" podUID="8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9" containerName="istio-proxy" containerID="cri-o://bebd93a441dd346b3ad67f75f901e29628a42001f82a11f6f733e61424ea621f" gracePeriod=30
Nov 03 00:41:27.709952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:27.709035    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:41:27.770955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:27.768351    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-podinfo\") pod \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") "
Nov 03 00:41:27.770955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:27.768414    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-credential-socket\") pod \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") "
Nov 03 00:41:27.770955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:27.768605    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-kube-api-access-8797w\" (UniqueName: \"kubernetes.io/projected/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-kube-api-access-8797w\") pod \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") "
Nov 03 00:41:27.770955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:27.768647    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-data\") pod \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") "
Nov 03 00:41:27.770955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:27.768690    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-envoy\") pod \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") "
Nov 03 00:41:27.770955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:27.768761    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-workload-socket\") pod \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") "
Nov 03 00:41:27.770955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:27.768812    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istiod-ca-cert\") pod \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") "
Nov 03 00:41:27.770955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:27.768839    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-workload-certs\") pod \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") "
Nov 03 00:41:27.971615 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:27.768897    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-token\" (UniqueName: \"kubernetes.io/projected/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-token\") pod \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\" (UID: \"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9\") "
Nov 03 00:41:28.049119 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.049065    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-token" pod "8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9" (UID: "8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:28.057551 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.057212    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-credential-socket" pod "8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9" (UID: "8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:28.075571 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.068801    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-workload-certs" pod "8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9" (UID: "8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:28.075571 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.069316    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istiod-ca-cert" pod "8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9" (UID: "8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:41:28.075571 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.069671    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-data" pod "8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9" (UID: "8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:28.075571 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.073052    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-workload-socket" pod "8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9" (UID: "8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:28.127673 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.127629    5012 generic.go:356] "Generic (PLEG): container finished" podID="8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9" containerID="bebd93a441dd346b3ad67f75f901e29628a42001f82a11f6f733e61424ea621f" exitCode=0
Nov 03 00:41:28.128004 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.127972    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6" event={"ID":"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9","Type":"ContainerDied","Data":"bebd93a441dd346b3ad67f75f901e29628a42001f82a11f6f733e61424ea621f"}
Nov 03 00:41:28.129029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.128066    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"
Nov 03 00:41:28.129029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.128239    5012 scope.go:122] "RemoveContainer" containerID="bebd93a441dd346b3ad67f75f901e29628a42001f82a11f6f733e61424ea621f"
Nov 03 00:41:28.129029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.128221    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6" event={"ID":"8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9","Type":"ContainerDied","Data":"2e05a4e17b07e5a6c3577643144970356b47b4df4c075bcb7ba726c7a53f9738"}
Nov 03 00:41:28.147370 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.133496    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-envoy" pod "8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9" (UID: "8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:28.147370 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.146189    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-podinfo" pod "8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9" (UID: "8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:41:28.147370 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.147314    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-kube-api-access-8797w" pod "8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9" (UID: "8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9"). InnerVolumeSpecName "kube-api-access-8797w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:28.147979 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.147773    5012 scope.go:122] "RemoveContainer" containerID="bebd93a441dd346b3ad67f75f901e29628a42001f82a11f6f733e61424ea621f"
Nov 03 00:41:28.148258 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:41:28.148227    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bebd93a441dd346b3ad67f75f901e29628a42001f82a11f6f733e61424ea621f\": container with ID starting with bebd93a441dd346b3ad67f75f901e29628a42001f82a11f6f733e61424ea621f not found: ID does not exist" containerID="bebd93a441dd346b3ad67f75f901e29628a42001f82a11f6f733e61424ea621f"
Nov 03 00:41:28.148362 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.148270    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bebd93a441dd346b3ad67f75f901e29628a42001f82a11f6f733e61424ea621f"} err="failed to get container status \"bebd93a441dd346b3ad67f75f901e29628a42001f82a11f6f733e61424ea621f\": rpc error: code = NotFound desc = could not find container \"bebd93a441dd346b3ad67f75f901e29628a42001f82a11f6f733e61424ea621f\": container with ID starting with bebd93a441dd346b3ad67f75f901e29628a42001f82a11f6f733e61424ea621f not found: ID does not exist"
Nov 03 00:41:28.172429 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.172393    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:28.172429 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.172429    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:28.172633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.172444    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8797w\" (UniqueName: \"kubernetes.io/projected/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-kube-api-access-8797w\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:28.172633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.172458    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:28.172633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.172469    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:28.172633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.172480    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:28.172633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.172492    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:28.172633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.172504    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:28.172633 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.172515    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:28.518931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.518007    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"]
Nov 03 00:41:28.522929 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:28.520936    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-lh6p6"]
Nov 03 00:41:29.248312 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:29.248272    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9" path="/var/lib/kubelet/pods/8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9/volumes"
Nov 03 00:41:30.131936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:30.127068    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl" podUID="2a1788b2-60c1-4e37-845b-879b4b5ef3f7" containerName="main" containerID="cri-o://443fa5965d757eac55a8efde91b1c00959aec45711ebb0350e568d86fbf79e86" gracePeriod=45
Nov 03 00:41:30.142331 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:30.142281    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl" podUID="2a1788b2-60c1-4e37-845b-879b4b5ef3f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.231:8000/ready\": EOF"
Nov 03 00:41:30.567582 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:30.567557    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:41:30.654777 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:30.654740    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-model-cache\") pod \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\" (UID: \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\") "
Nov 03 00:41:30.656580 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:30.656548    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-home\" (UniqueName: \"kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-home\") pod \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\" (UID: \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\") "
Nov 03 00:41:30.657105 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:30.657077    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-dshm\") pod \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\" (UID: \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\") "
Nov 03 00:41:30.657343 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:30.657325    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-tmp-dir\") pod \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\" (UID: \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\") "
Nov 03 00:41:30.657538 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:30.657521    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-kube-api-access-4rvkl\" (UniqueName: \"kubernetes.io/projected/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-kube-api-access-4rvkl\") pod \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\" (UID: \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\") "
Nov 03 00:41:30.657741 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:30.657725    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-tls-certs\") pod \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\" (UID: \"2a1788b2-60c1-4e37-845b-879b4b5ef3f7\") "
Nov 03 00:41:30.658711 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:30.656234    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-model-cache" pod "2a1788b2-60c1-4e37-845b-879b4b5ef3f7" (UID: "2a1788b2-60c1-4e37-845b-879b4b5ef3f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:30.658711 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:30.658188    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-tmp-dir" pod "2a1788b2-60c1-4e37-845b-879b4b5ef3f7" (UID: "2a1788b2-60c1-4e37-845b-879b4b5ef3f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:30.658711 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:30.658393    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-home" pod "2a1788b2-60c1-4e37-845b-879b4b5ef3f7" (UID: "2a1788b2-60c1-4e37-845b-879b4b5ef3f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:30.658711 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:30.658547    5012 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-tmp-dir\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:30.658711 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:30.658577    5012 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-model-cache\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:30.662235 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:30.662184    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-kube-api-access-4rvkl" pod "2a1788b2-60c1-4e37-845b-879b4b5ef3f7" (UID: "2a1788b2-60c1-4e37-845b-879b4b5ef3f7"). InnerVolumeSpecName "kube-api-access-4rvkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:30.662519 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:30.662499    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-tls-certs" pod "2a1788b2-60c1-4e37-845b-879b4b5ef3f7" (UID: "2a1788b2-60c1-4e37-845b-879b4b5ef3f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:41:30.664005 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:30.663977    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-dshm" pod "2a1788b2-60c1-4e37-845b-879b4b5ef3f7" (UID: "2a1788b2-60c1-4e37-845b-879b4b5ef3f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:30.759744 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:30.759711    5012 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-dshm\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:30.759744 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:30.759743    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rvkl\" (UniqueName: \"kubernetes.io/projected/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-kube-api-access-4rvkl\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:30.759983 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:30.759754    5012 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-tls-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:30.759983 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:30.759762    5012 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a1788b2-60c1-4e37-845b-879b4b5ef3f7-home\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:31.169664 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:31.169630    5012 generic.go:356] "Generic (PLEG): container finished" podID="2a1788b2-60c1-4e37-845b-879b4b5ef3f7" containerID="443fa5965d757eac55a8efde91b1c00959aec45711ebb0350e568d86fbf79e86" exitCode=0
Nov 03 00:41:31.169882 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:31.169710    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"
Nov 03 00:41:31.169882 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:31.169850    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl" event={"ID":"2a1788b2-60c1-4e37-845b-879b4b5ef3f7","Type":"ContainerDied","Data":"443fa5965d757eac55a8efde91b1c00959aec45711ebb0350e568d86fbf79e86"}
Nov 03 00:41:31.169882 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:31.169878    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl" event={"ID":"2a1788b2-60c1-4e37-845b-879b4b5ef3f7","Type":"ContainerDied","Data":"2a3d252ddacb90da329dd831e27de6c7735893c1b82515940340cdab77eb8726"}
Nov 03 00:41:31.170094 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:31.169895    5012 scope.go:122] "RemoveContainer" containerID="443fa5965d757eac55a8efde91b1c00959aec45711ebb0350e568d86fbf79e86"
Nov 03 00:41:31.185515 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:31.185483    5012 scope.go:122] "RemoveContainer" containerID="443fa5965d757eac55a8efde91b1c00959aec45711ebb0350e568d86fbf79e86"
Nov 03 00:41:31.185782 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:41:31.185765    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"443fa5965d757eac55a8efde91b1c00959aec45711ebb0350e568d86fbf79e86\": container with ID starting with 443fa5965d757eac55a8efde91b1c00959aec45711ebb0350e568d86fbf79e86 not found: ID does not exist" containerID="443fa5965d757eac55a8efde91b1c00959aec45711ebb0350e568d86fbf79e86"
Nov 03 00:41:31.185853 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:31.185789    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443fa5965d757eac55a8efde91b1c00959aec45711ebb0350e568d86fbf79e86"} err="failed to get container status \"443fa5965d757eac55a8efde91b1c00959aec45711ebb0350e568d86fbf79e86\": rpc error: code = NotFound desc = could not find container \"443fa5965d757eac55a8efde91b1c00959aec45711ebb0350e568d86fbf79e86\": container with ID starting with 443fa5965d757eac55a8efde91b1c00959aec45711ebb0350e568d86fbf79e86 not found: ID does not exist"
Nov 03 00:41:31.249735 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:31.249695    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"]
Nov 03 00:41:31.249899 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:31.249745    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2858fb/auth-test-model-2858fb-kserve-7946c49589-fjnfl"]
Nov 03 00:41:33.248594 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:33.248564    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a1788b2-60c1-4e37-845b-879b4b5ef3f7" path="/var/lib/kubelet/pods/2a1788b2-60c1-4e37-845b-879b4b5ef3f7/volumes"
Nov 03 00:41:37.616468 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:41:37.365041    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:41:38.267971 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:38.267932    5012 generic.go:356] "Generic (PLEG): container finished" podID="5eaa89b0-054c-4c59-a4af-02eefaa98bd6" containerID="374383517c1fa174e486e662af1576622e10d8b832772519d0b6a999c47de62a" exitCode=0
Nov 03 00:41:38.267971 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:38.267942    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc" event={"ID":"5eaa89b0-054c-4c59-a4af-02eefaa98bd6","Type":"ContainerDied","Data":"374383517c1fa174e486e662af1576622e10d8b832772519d0b6a999c47de62a"}
Nov 03 00:41:38.268261 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:38.268246    5012 scope.go:122] "RemoveContainer" containerID="374383517c1fa174e486e662af1576622e10d8b832772519d0b6a999c47de62a"
Nov 03 00:41:38.269829 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:38.269798    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:41:38.269985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:38.269882    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:41:38.269985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:38.269959    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:41:38.270123 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:38.270063    5012 generic.go:356] "Generic (PLEG): container finished" podID="76d45dee-a04f-4d36-9496-6e6eef77b94a" containerID="b643215a0a0279fff23cb45def4f49d25a7d531f12a10a1bf8f698788b783a32" exitCode=0
Nov 03 00:41:38.270123 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:38.270104    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq" event={"ID":"76d45dee-a04f-4d36-9496-6e6eef77b94a","Type":"ContainerDied","Data":"b643215a0a0279fff23cb45def4f49d25a7d531f12a10a1bf8f698788b783a32"}
Nov 03 00:41:38.270386 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:38.270370    5012 scope.go:122] "RemoveContainer" containerID="b643215a0a0279fff23cb45def4f49d25a7d531f12a10a1bf8f698788b783a32"
Nov 03 00:41:38.271805 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:38.271779    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:41:38.271980 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:38.271849    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:41:38.271980 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:38.271901    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:41:38.898933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:38.886953    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:41:39.285592 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:39.285497    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq" event={"ID":"76d45dee-a04f-4d36-9496-6e6eef77b94a","Type":"ContainerStarted","Data":"d8dc42bd9980ad190890f74b528c6d9fe721e26dfd77ba7d38cc7215a6b241ac"}
Nov 03 00:41:39.316979 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:39.316813    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc" event={"ID":"5eaa89b0-054c-4c59-a4af-02eefaa98bd6","Type":"ContainerStarted","Data":"94636314e788d5896234e9d328ac185709a3285f597fe3b013e9b072cc181a29"}
Nov 03 00:41:39.549163 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:39.549076    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:41:39.550643 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:39.550618    5012 patch_prober.go:29] interesting pod/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Nov 03 00:41:39.550771 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:39.550663    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc" podUID="5eaa89b0-054c-4c59-a4af-02eefaa98bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Nov 03 00:41:39.639447 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:39.639411    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:41:39.640494 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:39.640467    5012 patch_prober.go:29] interesting pod/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 03 00:41:39.640691 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:39.640667    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq" podUID="76d45dee-a04f-4d36-9496-6e6eef77b94a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 03 00:41:40.549617 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:40.549581    5012 patch_prober.go:29] interesting pod/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Nov 03 00:41:40.550064 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:40.549635    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc" podUID="5eaa89b0-054c-4c59-a4af-02eefaa98bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Nov 03 00:41:40.640245 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:40.640207    5012 patch_prober.go:29] interesting pod/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 03 00:41:40.640424 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:40.640262    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq" podUID="76d45dee-a04f-4d36-9496-6e6eef77b94a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 03 00:41:41.549525 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:41.549489    5012 patch_prober.go:29] interesting pod/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Nov 03 00:41:41.549775 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:41.549752    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc" podUID="5eaa89b0-054c-4c59-a4af-02eefaa98bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Nov 03 00:41:41.639303 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:41.639267    5012 patch_prober.go:29] interesting pod/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 03 00:41:41.639303 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:41.639314    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq" podUID="76d45dee-a04f-4d36-9496-6e6eef77b94a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 03 00:41:42.559451 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:42.557304    5012 patch_prober.go:29] interesting pod/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Nov 03 00:41:42.559451 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:42.557371    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc" podUID="5eaa89b0-054c-4c59-a4af-02eefaa98bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Nov 03 00:41:42.642931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:42.642305    5012 patch_prober.go:29] interesting pod/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 03 00:41:42.642931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:42.642368    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq" podUID="76d45dee-a04f-4d36-9496-6e6eef77b94a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 03 00:41:43.549763 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:43.549725    5012 patch_prober.go:29] interesting pod/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Nov 03 00:41:43.550012 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:43.549782    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc" podUID="5eaa89b0-054c-4c59-a4af-02eefaa98bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Nov 03 00:41:43.639391 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:43.639352    5012 patch_prober.go:29] interesting pod/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 03 00:41:43.639946 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:43.639417    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq" podUID="76d45dee-a04f-4d36-9496-6e6eef77b94a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 03 00:41:44.311077 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:44.311039    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:41:44.644283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:44.644237    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:41:44.644796 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:44.644666    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:41:44.645373 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:44.645353    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 00:41:44.828149 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:44.828106    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:41:44.829725 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:44.829697    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 00:41:48.777359 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.777321    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"]
Nov 03 00:41:48.783123 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.783094    5012 cpu_manager.go:414] "containerMap: removing container" podUID="2a1788b2-60c1-4e37-845b-879b4b5ef3f7" containerName="main"
Nov 03 00:41:48.783306 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.783288    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1788b2-60c1-4e37-845b-879b4b5ef3f7" containerName="main"
Nov 03 00:41:48.783424 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.783412    5012 cpu_manager.go:414] "containerMap: removing container" podUID="8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9" containerName="istio-proxy"
Nov 03 00:41:48.783507 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.783497    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9" containerName="istio-proxy"
Nov 03 00:41:48.783629 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.783619    5012 cpu_manager.go:414] "containerMap: removing container" podUID="f996bace-902b-4e8d-95bc-43f248835e98" containerName="istio-proxy"
Nov 03 00:41:48.783703 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.783693    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f996bace-902b-4e8d-95bc-43f248835e98" containerName="istio-proxy"
Nov 03 00:41:48.788041 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.788016    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="f996bace-902b-4e8d-95bc-43f248835e98" containerName="istio-proxy"
Nov 03 00:41:48.788211 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.788197    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1788b2-60c1-4e37-845b-879b4b5ef3f7" containerName="main"
Nov 03 00:41:48.788319 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.788307    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c86d7ef-71cd-4712-b0ed-5b1e4c50d3b9" containerName="istio-proxy"
Nov 03 00:41:48.831940 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.827454    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"]
Nov 03 00:41:48.831940 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.827827    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.851826 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.851791    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-token\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.852126 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.851837    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-data\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.852126 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.851881    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-credential-socket\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.852126 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.851930    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg6t8\" (UniqueName: \"kubernetes.io/projected/0b949944-2ed2-4a75-82d1-a2e2854e3355-kube-api-access-dg6t8\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.852126 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.852033    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-podinfo\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.852126 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.852075    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-envoy\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.852441 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.852132    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-workload-socket\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.852441 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.852199    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b949944-2ed2-4a75-82d1-a2e2854e3355-istiod-ca-cert\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.852441 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.852248    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-workload-certs\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.938650 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.938615    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-fdacdb-openshift-default-dockercfg-2krd9\""
Nov 03 00:41:48.953141 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.953112    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-workload-certs\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.953274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.953152    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-token\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.953274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.953179    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-data\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.953274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.953202    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-credential-socket\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.953274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.953226    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg6t8\" (UniqueName: \"kubernetes.io/projected/0b949944-2ed2-4a75-82d1-a2e2854e3355-kube-api-access-dg6t8\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.953487 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.953280    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-podinfo\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.953487 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.953327    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-envoy\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.953487 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.953410    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-workload-socket\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.953661 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.953516    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b949944-2ed2-4a75-82d1-a2e2854e3355-istiod-ca-cert\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.953661 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.953563    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-workload-certs\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.953661 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.953639    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-credential-socket\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.953817 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.953732    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-data\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.953895 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.953871    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-workload-socket\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.954127 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.954093    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b949944-2ed2-4a75-82d1-a2e2854e3355-istiod-ca-cert\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.955780 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.955744    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-envoy\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.955950 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.955895    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-podinfo\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.974927 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.974884    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-token\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:48.975014 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:48.974991    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg6t8\" (UniqueName: \"kubernetes.io/projected/0b949944-2ed2-4a75-82d1-a2e2854e3355-kube-api-access-dg6t8\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:49.213564 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:49.213533    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:49.530549 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:41:49.530515    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b949944_2ed2_4a75_82d1_a2e2854e3355.slice/crio-fdd5474fb59464e05bf18938e34710ba4c07b4a58fadf551f2785e608c37713c WatchSource:0}: Error finding container fdd5474fb59464e05bf18938e34710ba4c07b4a58fadf551f2785e608c37713c: Status 404 returned error can't find the container with id fdd5474fb59464e05bf18938e34710ba4c07b4a58fadf551f2785e608c37713c
Nov 03 00:41:49.531008 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:49.530984    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"]
Nov 03 00:41:49.532998 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:49.532962    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:41:49.533080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:49.533031    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:41:49.533080 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:49.533061    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:41:50.434937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:50.434873    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh" event={"ID":"0b949944-2ed2-4a75-82d1-a2e2854e3355","Type":"ContainerStarted","Data":"5a6d945d7210626b56ed10c44ef4956471df7943dc71b7c7ef1499cf4c2b2e7b"}
Nov 03 00:41:50.434937 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:50.434938    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh" event={"ID":"0b949944-2ed2-4a75-82d1-a2e2854e3355","Type":"ContainerStarted","Data":"fdd5474fb59464e05bf18938e34710ba4c07b4a58fadf551f2785e608c37713c"}
Nov 03 00:41:50.484122 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:50.484075    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh" podStartSLOduration=2.484059945 podStartE2EDuration="2.484059945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:50.482711355 +0000 UTC m=+2193.362295423" watchObservedRunningTime="2026-11-03 00:41:50.484059945 +0000 UTC m=+2193.363643989"
Nov 03 00:41:51.214206 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:51.214172    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:51.219528 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:51.219504    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:51.414755 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:51.414722    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f"]
Nov 03 00:41:51.419153 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:51.419132    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f"
Nov 03 00:41:51.437866 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:51.437834    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-d87781-serving-cert\""
Nov 03 00:41:51.439369 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:51.439343    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f"]
Nov 03 00:41:51.444856 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:51.444831    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:51.454169 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:51.454146    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:41:51.486094 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:51.486026    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f\" (UID: \"ce10e6fa-6b29-4edc-ac96-e81e7c7eebab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f"
Nov 03 00:41:51.486423 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:51.486392    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-metrics-certs\") pod \"maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f\" (UID: \"ce10e6fa-6b29-4edc-ac96-e81e7c7eebab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f"
Nov 03 00:41:51.486559 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:51.486500    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6h98\" (UniqueName: \"kubernetes.io/projected/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-kube-api-access-d6h98\") pod \"maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f\" (UID: \"ce10e6fa-6b29-4edc-ac96-e81e7c7eebab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f"
Nov 03 00:41:51.486559 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:51.486528    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f\" (UID: \"ce10e6fa-6b29-4edc-ac96-e81e7c7eebab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f"
Nov 03 00:41:51.587590 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:51.587553    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-metrics-certs\") pod \"maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f\" (UID: \"ce10e6fa-6b29-4edc-ac96-e81e7c7eebab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f"
Nov 03 00:41:51.587787 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:51.587612    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6h98\" (UniqueName: \"kubernetes.io/projected/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-kube-api-access-d6h98\") pod \"maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f\" (UID: \"ce10e6fa-6b29-4edc-ac96-e81e7c7eebab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f"
Nov 03 00:41:51.587787 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:51.587735    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f\" (UID: \"ce10e6fa-6b29-4edc-ac96-e81e7c7eebab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f"
Nov 03 00:41:51.587974 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:51.587803    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f\" (UID: \"ce10e6fa-6b29-4edc-ac96-e81e7c7eebab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f"
Nov 03 00:41:51.587974 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:41:51.587852    5012 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-serving-cert: secret "maas-api-e2e-extmod-a-d87781-serving-cert" not found
Nov 03 00:41:51.587974 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:41:51.587948    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-maas-api-tls podName:ce10e6fa-6b29-4edc-ac96-e81e7c7eebab nodeName:}" failed. No retries permitted until 2026-11-03 00:41:52.087920744 +0000 UTC m=+2194.967504793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-maas-api-tls") pod "maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f" (UID: "ce10e6fa-6b29-4edc-ac96-e81e7c7eebab") : secret "maas-api-e2e-extmod-a-d87781-serving-cert" not found
Nov 03 00:41:51.588489 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:51.588468    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f\" (UID: \"ce10e6fa-6b29-4edc-ac96-e81e7c7eebab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f"
Nov 03 00:41:51.590157 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:51.590133    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-metrics-certs\") pod \"maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f\" (UID: \"ce10e6fa-6b29-4edc-ac96-e81e7c7eebab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f"
Nov 03 00:41:51.605699 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:51.605675    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6h98\" (UniqueName: \"kubernetes.io/projected/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-kube-api-access-d6h98\") pod \"maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f\" (UID: \"ce10e6fa-6b29-4edc-ac96-e81e7c7eebab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f"
Nov 03 00:41:52.094388 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:52.094356    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f\" (UID: \"ce10e6fa-6b29-4edc-ac96-e81e7c7eebab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f"
Nov 03 00:41:52.096594 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:52.096572    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f\" (UID: \"ce10e6fa-6b29-4edc-ac96-e81e7c7eebab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f"
Nov 03 00:41:52.107593 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:52.107563    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"]
Nov 03 00:41:52.107845 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:52.107818    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9" podUID="81f130bb-d23a-4a79-b458-86a198845567" containerName="istio-proxy" containerID="cri-o://5bea062f3bbac1d808fc134907b4ea491b1005bdf592c94134baec8276833f98" gracePeriod=30
Nov 03 00:41:52.112298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:52.112275    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"]
Nov 03 00:41:52.112480 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:52.112459    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2" podUID="aa430510-e80c-4743-b4f3-838ce423e442" containerName="istio-proxy" containerID="cri-o://521e4c4e5517f7b94f7342f12d04e7dd45fa3017b977625cf1cb8ea97a4e08e0" gracePeriod=30
Nov 03 00:41:52.208759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:52.208725    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"]
Nov 03 00:41:52.208949 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:52.208928    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8" podUID="a0ee7800-4f9e-4307-8043-3d10c5bde48b" containerName="istio-proxy" containerID="cri-o://1143bb2983498e9445cd2a2c11ea90f0243cbecb2de10b263227c74308949ced" gracePeriod=30
Nov 03 00:41:52.335495 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:52.335453    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f"
Nov 03 00:41:52.668750 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:52.668717    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f"]
Nov 03 00:41:52.669252 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:41:52.668765    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce10e6fa_6b29_4edc_ac96_e81e7c7eebab.slice/crio-bdd8addd757a49ea59b4e6835843fd92b1811e19b1325e700645621a3aebd478 WatchSource:0}: Error finding container bdd8addd757a49ea59b4e6835843fd92b1811e19b1325e700645621a3aebd478: Status 404 returned error can't find the container with id bdd8addd757a49ea59b4e6835843fd92b1811e19b1325e700645621a3aebd478
Nov 03 00:41:53.492229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:53.492128    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f" event={"ID":"ce10e6fa-6b29-4edc-ac96-e81e7c7eebab","Type":"ContainerStarted","Data":"c8273d6e9ba441aeb7b8203f44632a940a1b17b8b0844d66c80fd868bd269bd2"}
Nov 03 00:41:53.492229 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:53.492196    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f" event={"ID":"ce10e6fa-6b29-4edc-ac96-e81e7c7eebab","Type":"ContainerStarted","Data":"bdd8addd757a49ea59b4e6835843fd92b1811e19b1325e700645621a3aebd478"}
Nov 03 00:41:53.492463 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:53.492445    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f"
Nov 03 00:41:53.529441 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:53.529388    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f" podStartSLOduration=2.070852387 podStartE2EDuration="2.529365523s" totalImagesPullingTime="458.513136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:53.527670145 +0000 UTC m=+2196.407254188" watchObservedRunningTime="2026-11-03 00:41:53.529365523 +0000 UTC m=+2196.408949566"
Nov 03 00:41:55.500591 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:55.500562    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb"]
Nov 03 00:41:55.505467 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:55.505442    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb"
Nov 03 00:41:55.520759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:55.519813    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb\" (UID: \"7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb"
Nov 03 00:41:55.520759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:55.519937    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbbcc\" (UniqueName: \"kubernetes.io/projected/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-kube-api-access-kbbcc\") pod \"maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb\" (UID: \"7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb"
Nov 03 00:41:55.520759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:55.520072    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb\" (UID: \"7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb"
Nov 03 00:41:55.520759 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:55.520131    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb\" (UID: \"7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb"
Nov 03 00:41:55.529268 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:55.529242    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-5c109b-serving-cert\""
Nov 03 00:41:55.536815 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:55.536786    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb"]
Nov 03 00:41:55.621039 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:55.621008    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbbcc\" (UniqueName: \"kubernetes.io/projected/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-kube-api-access-kbbcc\") pod \"maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb\" (UID: \"7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb"
Nov 03 00:41:55.621164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:55.621076    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb\" (UID: \"7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb"
Nov 03 00:41:55.621164 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:55.621111    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb\" (UID: \"7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb"
Nov 03 00:41:55.621285 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:55.621209    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb\" (UID: \"7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb"
Nov 03 00:41:55.621747 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:55.621724    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb\" (UID: \"7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb"
Nov 03 00:41:55.623703 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:55.623676    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb\" (UID: \"7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb"
Nov 03 00:41:55.624012 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:55.623990    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb\" (UID: \"7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb"
Nov 03 00:41:55.637526 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:55.637505    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbbcc\" (UniqueName: \"kubernetes.io/projected/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-kube-api-access-kbbcc\") pod \"maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb\" (UID: \"7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb"
Nov 03 00:41:55.853440 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:55.852242    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb"
Nov 03 00:41:56.260194 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:41:56.260163    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a4f41bb_b2f7_46d2_95e5_a265a55bdb0e.slice/crio-9818293a83a2c0bdf53dff24fb965ada9ce248eec51cf5e88b126176fd5191cf WatchSource:0}: Error finding container 9818293a83a2c0bdf53dff24fb965ada9ce248eec51cf5e88b126176fd5191cf: Status 404 returned error can't find the container with id 9818293a83a2c0bdf53dff24fb965ada9ce248eec51cf5e88b126176fd5191cf
Nov 03 00:41:56.260421 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:56.260401    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb"]
Nov 03 00:41:56.517206 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:56.517129    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb" event={"ID":"7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e","Type":"ContainerStarted","Data":"9818293a83a2c0bdf53dff24fb965ada9ce248eec51cf5e88b126176fd5191cf"}
Nov 03 00:41:56.783421 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:56.783326    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:41:56.783421 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:56.783386    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9" podUID="81f130bb-d23a-4a79-b458-86a198845567" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:41:57.401323 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.401291    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:41:57.452605 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.452575    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:41:57.543121 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.543085    5012 generic.go:356] "Generic (PLEG): container finished" podID="aa430510-e80c-4743-b4f3-838ce423e442" containerID="521e4c4e5517f7b94f7342f12d04e7dd45fa3017b977625cf1cb8ea97a4e08e0" exitCode=0
Nov 03 00:41:57.543552 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.543151    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"
Nov 03 00:41:57.544040 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.543169    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2" event={"ID":"aa430510-e80c-4743-b4f3-838ce423e442","Type":"ContainerDied","Data":"521e4c4e5517f7b94f7342f12d04e7dd45fa3017b977625cf1cb8ea97a4e08e0"}
Nov 03 00:41:57.544040 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.544019    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2" event={"ID":"aa430510-e80c-4743-b4f3-838ce423e442","Type":"ContainerDied","Data":"0f5a922ec1e8c60a53cd5cda0fc60e5cfd1a26dc861cf2596d498bb5502c6776"}
Nov 03 00:41:57.544223 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.544063    5012 scope.go:122] "RemoveContainer" containerID="521e4c4e5517f7b94f7342f12d04e7dd45fa3017b977625cf1cb8ea97a4e08e0"
Nov 03 00:41:57.546617 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.546591    5012 generic.go:356] "Generic (PLEG): container finished" podID="81f130bb-d23a-4a79-b458-86a198845567" containerID="5bea062f3bbac1d808fc134907b4ea491b1005bdf592c94134baec8276833f98" exitCode=0
Nov 03 00:41:57.546862 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.546839    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"
Nov 03 00:41:57.547049 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.547026    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9" event={"ID":"81f130bb-d23a-4a79-b458-86a198845567","Type":"ContainerDied","Data":"5bea062f3bbac1d808fc134907b4ea491b1005bdf592c94134baec8276833f98"}
Nov 03 00:41:57.547160 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.547073    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9" event={"ID":"81f130bb-d23a-4a79-b458-86a198845567","Type":"ContainerDied","Data":"b9e5e09089ee33999f8c1b94a54606d57b6310a2584789b686c321c9bf6147d2"}
Nov 03 00:41:57.547529 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.547445    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:41:57.550056 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.550007    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb" event={"ID":"7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e","Type":"ContainerStarted","Data":"724cbab7237ae96b722e68e586db350e17b43ebc2d7cb003928ad264c04618eb"}
Nov 03 00:41:57.550221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.550204    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb"
Nov 03 00:41:57.552489 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.552466    5012 generic.go:356] "Generic (PLEG): container finished" podID="a0ee7800-4f9e-4307-8043-3d10c5bde48b" containerID="1143bb2983498e9445cd2a2c11ea90f0243cbecb2de10b263227c74308949ced" exitCode=0
Nov 03 00:41:57.552663 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.552643    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8" event={"ID":"a0ee7800-4f9e-4307-8043-3d10c5bde48b","Type":"ContainerDied","Data":"1143bb2983498e9445cd2a2c11ea90f0243cbecb2de10b263227c74308949ced"}
Nov 03 00:41:57.553055 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.552723    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"
Nov 03 00:41:57.560700 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.560677    5012 scope.go:122] "RemoveContainer" containerID="521e4c4e5517f7b94f7342f12d04e7dd45fa3017b977625cf1cb8ea97a4e08e0"
Nov 03 00:41:57.561054 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:41:57.561028    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"521e4c4e5517f7b94f7342f12d04e7dd45fa3017b977625cf1cb8ea97a4e08e0\": container with ID starting with 521e4c4e5517f7b94f7342f12d04e7dd45fa3017b977625cf1cb8ea97a4e08e0 not found: ID does not exist" containerID="521e4c4e5517f7b94f7342f12d04e7dd45fa3017b977625cf1cb8ea97a4e08e0"
Nov 03 00:41:57.561147 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.561066    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521e4c4e5517f7b94f7342f12d04e7dd45fa3017b977625cf1cb8ea97a4e08e0"} err="failed to get container status \"521e4c4e5517f7b94f7342f12d04e7dd45fa3017b977625cf1cb8ea97a4e08e0\": rpc error: code = NotFound desc = could not find container \"521e4c4e5517f7b94f7342f12d04e7dd45fa3017b977625cf1cb8ea97a4e08e0\": container with ID starting with 521e4c4e5517f7b94f7342f12d04e7dd45fa3017b977625cf1cb8ea97a4e08e0 not found: ID does not exist"
Nov 03 00:41:57.561147 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.561089    5012 scope.go:122] "RemoveContainer" containerID="5bea062f3bbac1d808fc134907b4ea491b1005bdf592c94134baec8276833f98"
Nov 03 00:41:57.572646 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.572623    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-istio-data\") pod \"81f130bb-d23a-4a79-b458-86a198845567\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") "
Nov 03 00:41:57.572741 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.572668    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f130bb-d23a-4a79-b458-86a198845567-istio-token\" (UniqueName: \"kubernetes.io/projected/81f130bb-d23a-4a79-b458-86a198845567-istio-token\") pod \"81f130bb-d23a-4a79-b458-86a198845567\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") "
Nov 03 00:41:57.572741 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.572702    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-istio-data\") pod \"aa430510-e80c-4743-b4f3-838ce423e442\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") "
Nov 03 00:41:57.572919 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.572786    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-workload-socket\") pod \"81f130bb-d23a-4a79-b458-86a198845567\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") "
Nov 03 00:41:57.572990 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.572903    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-workload-certs\") pod \"aa430510-e80c-4743-b4f3-838ce423e442\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") "
Nov 03 00:41:57.573043 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.572987    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa430510-e80c-4743-b4f3-838ce423e442-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa430510-e80c-4743-b4f3-838ce423e442-istiod-ca-cert\") pod \"aa430510-e80c-4743-b4f3-838ce423e442\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") "
Nov 03 00:41:57.573043 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.572995    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-istio-data" pod "aa430510-e80c-4743-b4f3-838ce423e442" (UID: "aa430510-e80c-4743-b4f3-838ce423e442"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:57.573204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.573060    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-istio-envoy\") pod \"aa430510-e80c-4743-b4f3-838ce423e442\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") "
Nov 03 00:41:57.573204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.573074    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-workload-socket" pod "81f130bb-d23a-4a79-b458-86a198845567" (UID: "81f130bb-d23a-4a79-b458-86a198845567"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:57.573204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.573093    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa430510-e80c-4743-b4f3-838ce423e442-kube-api-access-4ggjn\" (UniqueName: \"kubernetes.io/projected/aa430510-e80c-4743-b4f3-838ce423e442-kube-api-access-4ggjn\") pod \"aa430510-e80c-4743-b4f3-838ce423e442\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") "
Nov 03 00:41:57.573204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.573108    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-istio-data" pod "81f130bb-d23a-4a79-b458-86a198845567" (UID: "81f130bb-d23a-4a79-b458-86a198845567"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:57.573204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.573124    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81f130bb-d23a-4a79-b458-86a198845567-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f130bb-d23a-4a79-b458-86a198845567-istio-podinfo\") pod \"81f130bb-d23a-4a79-b458-86a198845567\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") "
Nov 03 00:41:57.573204 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.573159    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-workload-certs\") pod \"81f130bb-d23a-4a79-b458-86a198845567\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") "
Nov 03 00:41:57.573527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.573208    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f130bb-d23a-4a79-b458-86a198845567-kube-api-access-j256j\" (UniqueName: \"kubernetes.io/projected/81f130bb-d23a-4a79-b458-86a198845567-kube-api-access-j256j\") pod \"81f130bb-d23a-4a79-b458-86a198845567\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") "
Nov 03 00:41:57.573527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.573306    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-istio-envoy\") pod \"81f130bb-d23a-4a79-b458-86a198845567\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") "
Nov 03 00:41:57.573527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.573316    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-workload-certs" pod "aa430510-e80c-4743-b4f3-838ce423e442" (UID: "aa430510-e80c-4743-b4f3-838ce423e442"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:57.573527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.573337    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81f130bb-d23a-4a79-b458-86a198845567-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f130bb-d23a-4a79-b458-86a198845567-istiod-ca-cert\") pod \"81f130bb-d23a-4a79-b458-86a198845567\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") "
Nov 03 00:41:57.573527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.573391    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-workload-socket\") pod \"aa430510-e80c-4743-b4f3-838ce423e442\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") "
Nov 03 00:41:57.573527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.573439    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa430510-e80c-4743-b4f3-838ce423e442-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa430510-e80c-4743-b4f3-838ce423e442-istio-podinfo\") pod \"aa430510-e80c-4743-b4f3-838ce423e442\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") "
Nov 03 00:41:57.573527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.573447    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-workload-certs" pod "81f130bb-d23a-4a79-b458-86a198845567" (UID: "81f130bb-d23a-4a79-b458-86a198845567"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:57.573527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.573508    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-credential-socket\") pod \"81f130bb-d23a-4a79-b458-86a198845567\" (UID: \"81f130bb-d23a-4a79-b458-86a198845567\") "
Nov 03 00:41:57.573527 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.573511    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa430510-e80c-4743-b4f3-838ce423e442-istiod-ca-cert" pod "aa430510-e80c-4743-b4f3-838ce423e442" (UID: "aa430510-e80c-4743-b4f3-838ce423e442"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:41:57.573993 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.573545    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa430510-e80c-4743-b4f3-838ce423e442-istio-token\" (UniqueName: \"kubernetes.io/projected/aa430510-e80c-4743-b4f3-838ce423e442-istio-token\") pod \"aa430510-e80c-4743-b4f3-838ce423e442\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") "
Nov 03 00:41:57.573993 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.573577    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-credential-socket\") pod \"aa430510-e80c-4743-b4f3-838ce423e442\" (UID: \"aa430510-e80c-4743-b4f3-838ce423e442\") "
Nov 03 00:41:57.573993 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.573737    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f130bb-d23a-4a79-b458-86a198845567-istiod-ca-cert" pod "81f130bb-d23a-4a79-b458-86a198845567" (UID: "81f130bb-d23a-4a79-b458-86a198845567"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:41:57.574272 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.574247    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f130bb-d23a-4a79-b458-86a198845567-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.574350 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.574276    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.574350 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.574289    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.574350 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.574303    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.574350 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.574317    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.574350 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.574331    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa430510-e80c-4743-b4f3-838ce423e442-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.574350 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.574343    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.574667 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.574588    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-credential-socket" pod "aa430510-e80c-4743-b4f3-838ce423e442" (UID: "aa430510-e80c-4743-b4f3-838ce423e442"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:57.574802 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.574781    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-credential-socket" pod "81f130bb-d23a-4a79-b458-86a198845567" (UID: "81f130bb-d23a-4a79-b458-86a198845567"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:57.575608 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.575581    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-workload-socket" pod "aa430510-e80c-4743-b4f3-838ce423e442" (UID: "aa430510-e80c-4743-b4f3-838ce423e442"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:57.576117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.576093    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa430510-e80c-4743-b4f3-838ce423e442-istio-podinfo" pod "aa430510-e80c-4743-b4f3-838ce423e442" (UID: "aa430510-e80c-4743-b4f3-838ce423e442"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:41:57.576260 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.576126    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f130bb-d23a-4a79-b458-86a198845567-kube-api-access-j256j" pod "81f130bb-d23a-4a79-b458-86a198845567" (UID: "81f130bb-d23a-4a79-b458-86a198845567"). InnerVolumeSpecName "kube-api-access-j256j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:57.576260 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.576185    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81f130bb-d23a-4a79-b458-86a198845567-istio-podinfo" pod "81f130bb-d23a-4a79-b458-86a198845567" (UID: "81f130bb-d23a-4a79-b458-86a198845567"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:41:57.576409 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.576352    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-istio-envoy" pod "81f130bb-d23a-4a79-b458-86a198845567" (UID: "81f130bb-d23a-4a79-b458-86a198845567"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:57.576513 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.576440    5012 scope.go:122] "RemoveContainer" containerID="5bea062f3bbac1d808fc134907b4ea491b1005bdf592c94134baec8276833f98"
Nov 03 00:41:57.576685 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:41:57.576666    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bea062f3bbac1d808fc134907b4ea491b1005bdf592c94134baec8276833f98\": container with ID starting with 5bea062f3bbac1d808fc134907b4ea491b1005bdf592c94134baec8276833f98 not found: ID does not exist" containerID="5bea062f3bbac1d808fc134907b4ea491b1005bdf592c94134baec8276833f98"
Nov 03 00:41:57.576760 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.576694    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bea062f3bbac1d808fc134907b4ea491b1005bdf592c94134baec8276833f98"} err="failed to get container status \"5bea062f3bbac1d808fc134907b4ea491b1005bdf592c94134baec8276833f98\": rpc error: code = NotFound desc = could not find container \"5bea062f3bbac1d808fc134907b4ea491b1005bdf592c94134baec8276833f98\": container with ID starting with 5bea062f3bbac1d808fc134907b4ea491b1005bdf592c94134baec8276833f98 not found: ID does not exist"
Nov 03 00:41:57.576760 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.576718    5012 scope.go:122] "RemoveContainer" containerID="1143bb2983498e9445cd2a2c11ea90f0243cbecb2de10b263227c74308949ced"
Nov 03 00:41:57.576952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.576901    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-istio-envoy" pod "aa430510-e80c-4743-b4f3-838ce423e442" (UID: "aa430510-e80c-4743-b4f3-838ce423e442"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:57.577639 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.577616    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa430510-e80c-4743-b4f3-838ce423e442-kube-api-access-4ggjn" pod "aa430510-e80c-4743-b4f3-838ce423e442" (UID: "aa430510-e80c-4743-b4f3-838ce423e442"). InnerVolumeSpecName "kube-api-access-4ggjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:57.578114 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.578090    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f130bb-d23a-4a79-b458-86a198845567-istio-token" pod "81f130bb-d23a-4a79-b458-86a198845567" (UID: "81f130bb-d23a-4a79-b458-86a198845567"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:57.578462 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.578444    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa430510-e80c-4743-b4f3-838ce423e442-istio-token" pod "aa430510-e80c-4743-b4f3-838ce423e442" (UID: "aa430510-e80c-4743-b4f3-838ce423e442"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:57.675817 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.674888    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-podinfo\") pod \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") "
Nov 03 00:41:57.675817 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.674998    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-envoy\") pod \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") "
Nov 03 00:41:57.675817 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.675048    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-token\" (UniqueName: \"kubernetes.io/projected/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-token\") pod \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") "
Nov 03 00:41:57.675817 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.675119    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-workload-certs\") pod \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") "
Nov 03 00:41:57.675817 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.675174    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0ee7800-4f9e-4307-8043-3d10c5bde48b-kube-api-access-flv42\" (UniqueName: \"kubernetes.io/projected/a0ee7800-4f9e-4307-8043-3d10c5bde48b-kube-api-access-flv42\") pod \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") "
Nov 03 00:41:57.675817 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.675229    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-credential-socket\") pod \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") "
Nov 03 00:41:57.675817 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.675296    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istiod-ca-cert\") pod \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") "
Nov 03 00:41:57.675817 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.675381    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-workload-socket\") pod \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") "
Nov 03 00:41:57.675817 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.675487    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-data\") pod \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\" (UID: \"a0ee7800-4f9e-4307-8043-3d10c5bde48b\") "
Nov 03 00:41:57.675817 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.675603    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-credential-socket" pod "a0ee7800-4f9e-4307-8043-3d10c5bde48b" (UID: "a0ee7800-4f9e-4307-8043-3d10c5bde48b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:57.676514 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.675842    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-workload-certs" pod "a0ee7800-4f9e-4307-8043-3d10c5bde48b" (UID: "a0ee7800-4f9e-4307-8043-3d10c5bde48b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:57.676514 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.676088    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-workload-socket" pod "a0ee7800-4f9e-4307-8043-3d10c5bde48b" (UID: "a0ee7800-4f9e-4307-8043-3d10c5bde48b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:57.676514 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.676297    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f130bb-d23a-4a79-b458-86a198845567-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.676514 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.676322    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.676514 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.676338    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.676514 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.676353    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.676514 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.676366    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ggjn\" (UniqueName: \"kubernetes.io/projected/aa430510-e80c-4743-b4f3-838ce423e442-kube-api-access-4ggjn\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.676514 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.676380    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f130bb-d23a-4a79-b458-86a198845567-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.676514 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.676394    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j256j\" (UniqueName: \"kubernetes.io/projected/81f130bb-d23a-4a79-b458-86a198845567-kube-api-access-j256j\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.676514 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.676407    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.676514 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.676421    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.676514 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.676435    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa430510-e80c-4743-b4f3-838ce423e442-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.676514 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.676449    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f130bb-d23a-4a79-b458-86a198845567-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.676514 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.676461    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa430510-e80c-4743-b4f3-838ce423e442-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.676514 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.676477    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa430510-e80c-4743-b4f3-838ce423e442-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.676514 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.676491    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.677221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.676596    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istiod-ca-cert" pod "a0ee7800-4f9e-4307-8043-3d10c5bde48b" (UID: "a0ee7800-4f9e-4307-8043-3d10c5bde48b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:41:57.677221 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.677182    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-data" pod "a0ee7800-4f9e-4307-8043-3d10c5bde48b" (UID: "a0ee7800-4f9e-4307-8043-3d10c5bde48b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:57.678995 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.678945    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-token" pod "a0ee7800-4f9e-4307-8043-3d10c5bde48b" (UID: "a0ee7800-4f9e-4307-8043-3d10c5bde48b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:57.679619 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.679572    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-podinfo" pod "a0ee7800-4f9e-4307-8043-3d10c5bde48b" (UID: "a0ee7800-4f9e-4307-8043-3d10c5bde48b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:41:57.681333 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.681306    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ee7800-4f9e-4307-8043-3d10c5bde48b-kube-api-access-flv42" pod "a0ee7800-4f9e-4307-8043-3d10c5bde48b" (UID: "a0ee7800-4f9e-4307-8043-3d10c5bde48b"). InnerVolumeSpecName "kube-api-access-flv42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:57.684049 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.684019    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-envoy" pod "a0ee7800-4f9e-4307-8043-3d10c5bde48b" (UID: "a0ee7800-4f9e-4307-8043-3d10c5bde48b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:57.730318 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.730267    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb" podStartSLOduration=2.394628026 podStartE2EDuration="2.730249061s" totalImagesPullingTime="335.621035ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:57.725641472 +0000 UTC m=+2200.605225516" watchObservedRunningTime="2026-11-03 00:41:57.730249061 +0000 UTC m=+2200.609833104"
Nov 03 00:41:57.778158 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.778128    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.778158 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.778158    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.778158 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.778166    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.778412 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.778178    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.778412 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.778192    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0ee7800-4f9e-4307-8043-3d10c5bde48b-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.778412 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.778204    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flv42\" (UniqueName: \"kubernetes.io/projected/a0ee7800-4f9e-4307-8043-3d10c5bde48b-kube-api-access-flv42\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.893326 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.893251    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"]
Nov 03 00:41:57.899508 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.899472    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-5lnf2"]
Nov 03 00:41:57.997981 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:57.991003    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"]
Nov 03 00:41:58.081548 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:58.081510    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zmcw9"]
Nov 03 00:41:58.088492 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:58.088464    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"]
Nov 03 00:41:58.092952 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:58.092920    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-wn5n8"]
Nov 03 00:41:59.245034 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:59.244999    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81f130bb-d23a-4a79-b458-86a198845567" path="/var/lib/kubelet/pods/81f130bb-d23a-4a79-b458-86a198845567/volumes"
Nov 03 00:41:59.245685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:59.245662    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0ee7800-4f9e-4307-8043-3d10c5bde48b" path="/var/lib/kubelet/pods/a0ee7800-4f9e-4307-8043-3d10c5bde48b/volumes"
Nov 03 00:41:59.246222 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:59.246201    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa430510-e80c-4743-b4f3-838ce423e442" path="/var/lib/kubelet/pods/aa430510-e80c-4743-b4f3-838ce423e442/volumes"
Nov 03 00:41:59.506085 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:41:59.506015    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f"
Nov 03 00:42:03.569102 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:03.569075    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb"
Nov 03 00:42:07.398466 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:42:07.398413    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:42:25.122551 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.122519    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7"]
Nov 03 00:42:25.123227 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.123209    5012 cpu_manager.go:414] "containerMap: removing container" podUID="a0ee7800-4f9e-4307-8043-3d10c5bde48b" containerName="istio-proxy"
Nov 03 00:42:25.123281 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.123228    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0ee7800-4f9e-4307-8043-3d10c5bde48b" containerName="istio-proxy"
Nov 03 00:42:25.123281 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.123244    5012 cpu_manager.go:414] "containerMap: removing container" podUID="81f130bb-d23a-4a79-b458-86a198845567" containerName="istio-proxy"
Nov 03 00:42:25.123281 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.123250    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f130bb-d23a-4a79-b458-86a198845567" containerName="istio-proxy"
Nov 03 00:42:25.123389 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.123286    5012 cpu_manager.go:414] "containerMap: removing container" podUID="aa430510-e80c-4743-b4f3-838ce423e442" containerName="istio-proxy"
Nov 03 00:42:25.123389 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.123291    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa430510-e80c-4743-b4f3-838ce423e442" containerName="istio-proxy"
Nov 03 00:42:25.123455 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.123390    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa430510-e80c-4743-b4f3-838ce423e442" containerName="istio-proxy"
Nov 03 00:42:25.123455 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.123403    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0ee7800-4f9e-4307-8043-3d10c5bde48b" containerName="istio-proxy"
Nov 03 00:42:25.123455 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.123412    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f130bb-d23a-4a79-b458-86a198845567" containerName="istio-proxy"
Nov 03 00:42:25.128460 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.128443    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7"
Nov 03 00:42:25.149123 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.149100    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7"]
Nov 03 00:42:25.154981 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.154950    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-fdacdb-serving-cert\""
Nov 03 00:42:25.236443 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.236409    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7\" (UID: \"2c06dcd4-76c3-4d89-85cd-d14ee32d6567\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7"
Nov 03 00:42:25.236630 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.236469    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqdnw\" (UniqueName: \"kubernetes.io/projected/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-kube-api-access-gqdnw\") pod \"maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7\" (UID: \"2c06dcd4-76c3-4d89-85cd-d14ee32d6567\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7"
Nov 03 00:42:25.236709 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.236639    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7\" (UID: \"2c06dcd4-76c3-4d89-85cd-d14ee32d6567\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7"
Nov 03 00:42:25.236859 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.236840    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-metrics-certs\") pod \"maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7\" (UID: \"2c06dcd4-76c3-4d89-85cd-d14ee32d6567\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7"
Nov 03 00:42:25.344400 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.344359    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7\" (UID: \"2c06dcd4-76c3-4d89-85cd-d14ee32d6567\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7"
Nov 03 00:42:25.344841 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.344810    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-metrics-certs\") pod \"maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7\" (UID: \"2c06dcd4-76c3-4d89-85cd-d14ee32d6567\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7"
Nov 03 00:42:25.345165 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.345142    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7\" (UID: \"2c06dcd4-76c3-4d89-85cd-d14ee32d6567\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7"
Nov 03 00:42:25.345372 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.345354    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqdnw\" (UniqueName: \"kubernetes.io/projected/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-kube-api-access-gqdnw\") pod \"maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7\" (UID: \"2c06dcd4-76c3-4d89-85cd-d14ee32d6567\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7"
Nov 03 00:42:25.347769 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:42:25.346607    5012 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-serving-cert: secret "maas-api-e2e-extmod-b-fdacdb-serving-cert" not found
Nov 03 00:42:25.347769 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:42:25.346674    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-maas-api-tls podName:2c06dcd4-76c3-4d89-85cd-d14ee32d6567 nodeName:}" failed. No retries permitted until 2026-11-03 00:42:25.846652721 +0000 UTC m=+2228.726236761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-maas-api-tls") pod "maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7" (UID: "2c06dcd4-76c3-4d89-85cd-d14ee32d6567") : secret "maas-api-e2e-extmod-b-fdacdb-serving-cert" not found
Nov 03 00:42:25.353369 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.353345    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7\" (UID: \"2c06dcd4-76c3-4d89-85cd-d14ee32d6567\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7"
Nov 03 00:42:25.357254 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.357197    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-metrics-certs\") pod \"maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7\" (UID: \"2c06dcd4-76c3-4d89-85cd-d14ee32d6567\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7"
Nov 03 00:42:25.371758 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.371729    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqdnw\" (UniqueName: \"kubernetes.io/projected/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-kube-api-access-gqdnw\") pod \"maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7\" (UID: \"2c06dcd4-76c3-4d89-85cd-d14ee32d6567\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7"
Nov 03 00:42:25.851358 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.851325    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7\" (UID: \"2c06dcd4-76c3-4d89-85cd-d14ee32d6567\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7"
Nov 03 00:42:25.853835 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:25.853806    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7\" (UID: \"2c06dcd4-76c3-4d89-85cd-d14ee32d6567\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7"
Nov 03 00:42:26.042870 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:26.042844    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7"
Nov 03 00:42:26.345796 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:26.345767    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7"]
Nov 03 00:42:26.346186 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:42:26.345775    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c06dcd4_76c3_4d89_85cd_d14ee32d6567.slice/crio-3ddfbf0c7805452b628d87aa475445f4d8aeb524a90d09250b03c59e0bf20c33 WatchSource:0}: Error finding container 3ddfbf0c7805452b628d87aa475445f4d8aeb524a90d09250b03c59e0bf20c33: Status 404 returned error can't find the container with id 3ddfbf0c7805452b628d87aa475445f4d8aeb524a90d09250b03c59e0bf20c33
Nov 03 00:42:26.820538 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:26.820501    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7" event={"ID":"2c06dcd4-76c3-4d89-85cd-d14ee32d6567","Type":"ContainerStarted","Data":"7de4e5c5c51631b34c367eafb9829d00ae79b3979f19e20688f314559ae9eabc"}
Nov 03 00:42:26.820686 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:26.820560    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7" event={"ID":"2c06dcd4-76c3-4d89-85cd-d14ee32d6567","Type":"ContainerStarted","Data":"3ddfbf0c7805452b628d87aa475445f4d8aeb524a90d09250b03c59e0bf20c33"}
Nov 03 00:42:26.820686 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:26.820674    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7"
Nov 03 00:42:26.855634 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:26.855585    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7" podStartSLOduration=1.563770988 podStartE2EDuration="1.855570795s" totalImagesPullingTime="291.799807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:42:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:42:26.855363255 +0000 UTC m=+2229.734947297" watchObservedRunningTime="2026-11-03 00:42:26.855570795 +0000 UTC m=+2229.735154839"
Nov 03 00:42:32.833523 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:32.833485    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7"
Nov 03 00:42:37.364129 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:42:37.364090    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:42:49.018786 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:49.018680    5012 generic.go:356] "Generic (PLEG): container finished" podID="0b949944-2ed2-4a75-82d1-a2e2854e3355" containerID="5a6d945d7210626b56ed10c44ef4956471df7943dc71b7c7ef1499cf4c2b2e7b" exitCode=0
Nov 03 00:42:49.019347 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:49.018787    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh" event={"ID":"0b949944-2ed2-4a75-82d1-a2e2854e3355","Type":"ContainerDied","Data":"5a6d945d7210626b56ed10c44ef4956471df7943dc71b7c7ef1499cf4c2b2e7b"}
Nov 03 00:42:49.019347 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:49.019301    5012 scope.go:122] "RemoveContainer" containerID="5a6d945d7210626b56ed10c44ef4956471df7943dc71b7c7ef1499cf4c2b2e7b"
Nov 03 00:42:49.022013 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:49.021980    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:42:49.022135 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:49.022057    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:42:49.022135 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:49.022089    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:42:50.029253 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:50.029217    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh" event={"ID":"0b949944-2ed2-4a75-82d1-a2e2854e3355","Type":"ContainerStarted","Data":"73c20949c8b010fff7a490effee865357f1b0878eb699c1cf6ebe7e70dde9a1e"}
Nov 03 00:42:50.222948 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:50.222760    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:42:50.224660 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:50.224632    5012 patch_prober.go:29] interesting pod/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body=
Nov 03 00:42:50.224810 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:50.224701    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh" podUID="0b949944-2ed2-4a75-82d1-a2e2854e3355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused"
Nov 03 00:42:51.214375 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:51.214338    5012 patch_prober.go:29] interesting pod/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body=
Nov 03 00:42:51.215054 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:51.214391    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh" podUID="0b949944-2ed2-4a75-82d1-a2e2854e3355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused"
Nov 03 00:42:51.445470 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:51.445438    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:42:52.214403 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:52.214367    5012 patch_prober.go:29] interesting pod/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body=
Nov 03 00:42:52.214806 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:52.214424    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh" podUID="0b949944-2ed2-4a75-82d1-a2e2854e3355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused"
Nov 03 00:42:53.214478 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:53.214444    5012 patch_prober.go:29] interesting pod/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body=
Nov 03 00:42:53.214878 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:53.214506    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh" podUID="0b949944-2ed2-4a75-82d1-a2e2854e3355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused"
Nov 03 00:42:54.214347 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:54.214314    5012 patch_prober.go:29] interesting pod/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body=
Nov 03 00:42:54.214535 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:54.214377    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh" podUID="0b949944-2ed2-4a75-82d1-a2e2854e3355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused"
Nov 03 00:42:55.218438 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:55.218398    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:42:55.219557 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:42:55.219531    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 00:43:07.364751 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:43:07.364712    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:43:14.306791 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:14.306757    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk"]
Nov 03 00:43:14.311454 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:14.311437    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk"
Nov 03 00:43:14.329115 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:14.329090    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk"]
Nov 03 00:43:14.333966 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:14.333939    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-d4d8c2-serving-cert\""
Nov 03 00:43:14.461399 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:14.461364    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk\" (UID: \"2a72ca88-0371-4fd7-ab82-3019bd21d9f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk"
Nov 03 00:43:14.461569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:14.461498    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4zx7\" (UniqueName: \"kubernetes.io/projected/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-kube-api-access-j4zx7\") pod \"maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk\" (UID: \"2a72ca88-0371-4fd7-ab82-3019bd21d9f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk"
Nov 03 00:43:14.461626 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:14.461593    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk\" (UID: \"2a72ca88-0371-4fd7-ab82-3019bd21d9f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk"
Nov 03 00:43:14.461704 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:14.461682    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-metrics-certs\") pod \"maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk\" (UID: \"2a72ca88-0371-4fd7-ab82-3019bd21d9f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk"
Nov 03 00:43:14.562802 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:14.562721    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4zx7\" (UniqueName: \"kubernetes.io/projected/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-kube-api-access-j4zx7\") pod \"maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk\" (UID: \"2a72ca88-0371-4fd7-ab82-3019bd21d9f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk"
Nov 03 00:43:14.562994 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:14.562860    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk\" (UID: \"2a72ca88-0371-4fd7-ab82-3019bd21d9f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk"
Nov 03 00:43:14.562994 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:14.562902    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-metrics-certs\") pod \"maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk\" (UID: \"2a72ca88-0371-4fd7-ab82-3019bd21d9f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk"
Nov 03 00:43:14.562994 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:14.562975    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk\" (UID: \"2a72ca88-0371-4fd7-ab82-3019bd21d9f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk"
Nov 03 00:43:14.563197 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:43:14.563172    5012 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-serving-cert: secret "maas-api-e2e-extmod-b-d4d8c2-serving-cert" not found
Nov 03 00:43:14.563287 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:43:14.563267    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-maas-api-tls podName:2a72ca88-0371-4fd7-ab82-3019bd21d9f6 nodeName:}" failed. No retries permitted until 2026-11-03 00:43:15.063246367 +0000 UTC m=+2277.942830405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-maas-api-tls") pod "maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk" (UID: "2a72ca88-0371-4fd7-ab82-3019bd21d9f6") : secret "maas-api-e2e-extmod-b-d4d8c2-serving-cert" not found
Nov 03 00:43:14.563632 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:14.563609    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk\" (UID: \"2a72ca88-0371-4fd7-ab82-3019bd21d9f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk"
Nov 03 00:43:14.565899 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:14.565872    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-metrics-certs\") pod \"maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk\" (UID: \"2a72ca88-0371-4fd7-ab82-3019bd21d9f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk"
Nov 03 00:43:14.587276 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:14.587253    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4zx7\" (UniqueName: \"kubernetes.io/projected/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-kube-api-access-j4zx7\") pod \"maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk\" (UID: \"2a72ca88-0371-4fd7-ab82-3019bd21d9f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk"
Nov 03 00:43:15.068337 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:15.068303    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk\" (UID: \"2a72ca88-0371-4fd7-ab82-3019bd21d9f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk"
Nov 03 00:43:15.070817 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:15.070790    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk\" (UID: \"2a72ca88-0371-4fd7-ab82-3019bd21d9f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk"
Nov 03 00:43:15.225634 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:15.225601    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk"
Nov 03 00:43:15.532464 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:15.532437    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk"]
Nov 03 00:43:15.532961 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:43:15.532937    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a72ca88_0371_4fd7_ab82_3019bd21d9f6.slice/crio-42eefa4c7a59962ccc5f596b64b3209c0f0803663cb40c002a87b01565dc055d WatchSource:0}: Error finding container 42eefa4c7a59962ccc5f596b64b3209c0f0803663cb40c002a87b01565dc055d: Status 404 returned error can't find the container with id 42eefa4c7a59962ccc5f596b64b3209c0f0803663cb40c002a87b01565dc055d
Nov 03 00:43:16.291181 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:16.291149    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk" event={"ID":"2a72ca88-0371-4fd7-ab82-3019bd21d9f6","Type":"ContainerStarted","Data":"42eefa4c7a59962ccc5f596b64b3209c0f0803663cb40c002a87b01565dc055d"}
Nov 03 00:43:17.300014 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:17.299971    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk" event={"ID":"2a72ca88-0371-4fd7-ab82-3019bd21d9f6","Type":"ContainerStarted","Data":"02fb35688ea869c27c7723f531fe80e0411dff0f0da115c363fb005320116933"}
Nov 03 00:43:17.300407 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:17.300046    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk"
Nov 03 00:43:23.313021 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:23.312992    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk"
Nov 03 00:43:23.351567 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:23.351518    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk" podStartSLOduration=8.664105611 podStartE2EDuration="9.351504235s" totalImagesPullingTime="687.398624ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:43:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:43:17.347413597 +0000 UTC m=+2280.226997642" watchObservedRunningTime="2026-11-03 00:43:23.351504235 +0000 UTC m=+2286.231088315"
Nov 03 00:43:37.363052 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:43:37.362965    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:43:53.007058 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:53.007025    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c"]
Nov 03 00:43:53.011897 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:53.011876    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c"
Nov 03 00:43:53.024916 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:53.024881    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-850433-serving-cert\""
Nov 03 00:43:53.042948 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:53.042899    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c"]
Nov 03 00:43:53.148532 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:53.148493    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c\" (UID: \"1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c"
Nov 03 00:43:53.148718 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:53.148654    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c\" (UID: \"1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c"
Nov 03 00:43:53.148798 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:53.148755    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-metrics-certs\") pod \"maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c\" (UID: \"1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c"
Nov 03 00:43:53.148877 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:53.148863    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29m4k\" (UniqueName: \"kubernetes.io/projected/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-kube-api-access-29m4k\") pod \"maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c\" (UID: \"1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c"
Nov 03 00:43:53.250658 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:53.250621    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c\" (UID: \"1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c"
Nov 03 00:43:53.250866 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:53.250669    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-metrics-certs\") pod \"maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c\" (UID: \"1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c"
Nov 03 00:43:53.250866 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:53.250722    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29m4k\" (UniqueName: \"kubernetes.io/projected/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-kube-api-access-29m4k\") pod \"maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c\" (UID: \"1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c"
Nov 03 00:43:53.250866 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:53.250779    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c\" (UID: \"1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c"
Nov 03 00:43:53.250866 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:43:53.250788    5012 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-serving-cert: secret "maas-api-e2e-extmod-a-850433-serving-cert" not found
Nov 03 00:43:53.250866 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:43:53.250869    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-maas-api-tls podName:1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a nodeName:}" failed. No retries permitted until 2026-11-03 00:43:53.750849987 +0000 UTC m=+2316.630434022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-maas-api-tls") pod "maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c" (UID: "1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a") : secret "maas-api-e2e-extmod-a-850433-serving-cert" not found
Nov 03 00:43:53.251543 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:53.251354    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c\" (UID: \"1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c"
Nov 03 00:43:53.253733 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:53.253705    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-metrics-certs\") pod \"maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c\" (UID: \"1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c"
Nov 03 00:43:53.268718 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:53.268666    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29m4k\" (UniqueName: \"kubernetes.io/projected/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-kube-api-access-29m4k\") pod \"maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c\" (UID: \"1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c"
Nov 03 00:43:53.757318 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:53.757288    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c\" (UID: \"1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c"
Nov 03 00:43:53.759622 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:53.759592    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c\" (UID: \"1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c"
Nov 03 00:43:53.926124 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:53.926092    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c"
Nov 03 00:43:54.250645 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:54.243383    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c"]
Nov 03 00:43:54.634958 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:54.634925    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c" event={"ID":"1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a","Type":"ContainerStarted","Data":"1af5310ce3f0ee022062254ec75a540dfc508f048a87dda724439d26956a55fe"}
Nov 03 00:43:54.635087 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:54.634986    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c" event={"ID":"1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a","Type":"ContainerStarted","Data":"33018606f06f08a0b78ecc0d53528f65601e0d6b6ee31a49ab52db5f12691bb0"}
Nov 03 00:43:54.635087 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:54.635026    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c"
Nov 03 00:43:54.675818 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:43:54.675768    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c" podStartSLOduration=2.404506421 podStartE2EDuration="2.675753671s" totalImagesPullingTime="271.24725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:43:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:43:54.671223645 +0000 UTC m=+2317.550807688" watchObservedRunningTime="2026-11-03 00:43:54.675753671 +0000 UTC m=+2317.555337713"
Nov 03 00:44:00.648857 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:00.648821    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c"
Nov 03 00:44:07.363249 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:44:07.363210    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:44:13.988314 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:13.988275    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2"]
Nov 03 00:44:13.993070 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:13.993050    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2"
Nov 03 00:44:14.012372 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:14.012343    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2"]
Nov 03 00:44:14.066936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:14.059122    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-0db47e\""
Nov 03 00:44:14.082005 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:14.068215    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-0db47e-dockercfg-qqhvv\""
Nov 03 00:44:14.107586 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:14.106813    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwc9x\" (UniqueName: \"kubernetes.io/projected/a47c095a-1345-4bb2-a1db-b718dcf7261f-kube-api-access-lwc9x\") pod \"payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2\" (UID: \"a47c095a-1345-4bb2-a1db-b718dcf7261f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2"
Nov 03 00:44:14.107586 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:14.107018    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47c095a-1345-4bb2-a1db-b718dcf7261f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2\" (UID: \"a47c095a-1345-4bb2-a1db-b718dcf7261f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2"
Nov 03 00:44:14.209199 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:14.209147    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47c095a-1345-4bb2-a1db-b718dcf7261f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2\" (UID: \"a47c095a-1345-4bb2-a1db-b718dcf7261f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2"
Nov 03 00:44:14.209465 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:14.209440    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwc9x\" (UniqueName: \"kubernetes.io/projected/a47c095a-1345-4bb2-a1db-b718dcf7261f-kube-api-access-lwc9x\") pod \"payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2\" (UID: \"a47c095a-1345-4bb2-a1db-b718dcf7261f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2"
Nov 03 00:44:14.210098 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:14.210075    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47c095a-1345-4bb2-a1db-b718dcf7261f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2\" (UID: \"a47c095a-1345-4bb2-a1db-b718dcf7261f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2"
Nov 03 00:44:14.226733 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:14.226699    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwc9x\" (UniqueName: \"kubernetes.io/projected/a47c095a-1345-4bb2-a1db-b718dcf7261f-kube-api-access-lwc9x\") pod \"payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2\" (UID: \"a47c095a-1345-4bb2-a1db-b718dcf7261f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2"
Nov 03 00:44:14.307790 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:14.307727    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2"
Nov 03 00:44:14.610311 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:44:14.610279    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47c095a_1345_4bb2_a1db_b718dcf7261f.slice/crio-9730878c43971764aa3a3cc44417952c064125ffc8736cc7e0f0ef46cd794f2a WatchSource:0}: Error finding container 9730878c43971764aa3a3cc44417952c064125ffc8736cc7e0f0ef46cd794f2a: Status 404 returned error can't find the container with id 9730878c43971764aa3a3cc44417952c064125ffc8736cc7e0f0ef46cd794f2a
Nov 03 00:44:14.610657 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:14.610634    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2"]
Nov 03 00:44:14.861548 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:14.861471    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2" event={"ID":"a47c095a-1345-4bb2-a1db-b718dcf7261f","Type":"ContainerStarted","Data":"00f2ad9e5dd2396b090eb3259b66a189c64310f268b23da2d70b0f6549dd4e02"}
Nov 03 00:44:14.861548 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:14.861532    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2" event={"ID":"a47c095a-1345-4bb2-a1db-b718dcf7261f","Type":"ContainerStarted","Data":"9730878c43971764aa3a3cc44417952c064125ffc8736cc7e0f0ef46cd794f2a"}
Nov 03 00:44:14.861716 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:14.861558    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2"
Nov 03 00:44:14.900246 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:14.900192    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2" podStartSLOduration=1.900179916 podStartE2EDuration="1.900179916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:14.897716986 +0000 UTC m=+2337.777301029" watchObservedRunningTime="2026-11-03 00:44:14.900179916 +0000 UTC m=+2337.779763958"
Nov 03 00:44:18.119018 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:18.118983    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx"]
Nov 03 00:44:18.123688 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:18.123671    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx"
Nov 03 00:44:18.147767 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:18.147741    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-909268-serving-cert\""
Nov 03 00:44:18.153601 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:18.153571    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx"]
Nov 03 00:44:18.265498 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:18.265468    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcxft\" (UniqueName: \"kubernetes.io/projected/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-kube-api-access-wcxft\") pod \"maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx\" (UID: \"2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx"
Nov 03 00:44:18.265652 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:18.265603    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx\" (UID: \"2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx"
Nov 03 00:44:18.265703 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:18.265688    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-maas-api-tls\") pod \"maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx\" (UID: \"2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx"
Nov 03 00:44:18.265803 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:18.265786    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-metrics-certs\") pod \"maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx\" (UID: \"2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx"
Nov 03 00:44:18.367082 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:18.367044    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-maas-api-tls\") pod \"maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx\" (UID: \"2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx"
Nov 03 00:44:18.367259 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:44:18.367191    5012 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-serving-cert: secret "maas-api-e2e-extmod-b-909268-serving-cert" not found
Nov 03 00:44:18.367259 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:18.367196    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-metrics-certs\") pod \"maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx\" (UID: \"2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx"
Nov 03 00:44:18.367372 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:44:18.367254    5012 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-maas-api-tls podName:2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64 nodeName:}" failed. No retries permitted until 2026-11-03 00:44:18.867237911 +0000 UTC m=+2341.746821932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-maas-api-tls") pod "maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx" (UID: "2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64") : secret "maas-api-e2e-extmod-b-909268-serving-cert" not found
Nov 03 00:44:18.367419 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:18.367384    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcxft\" (UniqueName: \"kubernetes.io/projected/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-kube-api-access-wcxft\") pod \"maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx\" (UID: \"2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx"
Nov 03 00:44:18.367504 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:18.367488    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx\" (UID: \"2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx"
Nov 03 00:44:18.368244 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:18.368219    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx\" (UID: \"2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx"
Nov 03 00:44:18.369855 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:18.369805    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-metrics-certs\") pod \"maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx\" (UID: \"2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx"
Nov 03 00:44:18.385068 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:18.385047    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcxft\" (UniqueName: \"kubernetes.io/projected/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-kube-api-access-wcxft\") pod \"maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx\" (UID: \"2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx"
Nov 03 00:44:18.873321 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:18.873280    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-maas-api-tls\") pod \"maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx\" (UID: \"2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx"
Nov 03 00:44:18.875803 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:18.875779    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-maas-api-tls\") pod \"maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx\" (UID: \"2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx"
Nov 03 00:44:19.038006 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:19.037965    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx"
Nov 03 00:44:19.418526 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:19.418477    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx"]
Nov 03 00:44:19.418991 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:44:19.418967    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4e5fb2_d1a9_4070_8e9b_3c60656bfa64.slice/crio-acc9bfc32704d824872fb1dc126be9482642a2b98fe26dfee0adb55bda78bcb5 WatchSource:0}: Error finding container acc9bfc32704d824872fb1dc126be9482642a2b98fe26dfee0adb55bda78bcb5: Status 404 returned error can't find the container with id acc9bfc32704d824872fb1dc126be9482642a2b98fe26dfee0adb55bda78bcb5
Nov 03 00:44:19.904553 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:19.904514    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx" event={"ID":"2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64","Type":"ContainerStarted","Data":"971b09c4aa28d778da4894db6dbb2cf297f826c8efb336623357b4858990a921"}
Nov 03 00:44:19.904703 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:19.904567    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx" event={"ID":"2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64","Type":"ContainerStarted","Data":"acc9bfc32704d824872fb1dc126be9482642a2b98fe26dfee0adb55bda78bcb5"}
Nov 03 00:44:19.904703 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:19.904603    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx"
Nov 03 00:44:19.941510 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:19.941468    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx" podStartSLOduration=1.581727371 podStartE2EDuration="1.941456117s" totalImagesPullingTime="359.728746ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:19.93798931 +0000 UTC m=+2342.817573352" watchObservedRunningTime="2026-11-03 00:44:19.941456117 +0000 UTC m=+2342.821040159"
Nov 03 00:44:25.872459 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:25.872429    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2"
Nov 03 00:44:25.930981 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:25.930948    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx"
Nov 03 00:44:37.363937 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:44:37.363883    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:44:53.474948 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:53.474896    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg"]
Nov 03 00:44:53.480715 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:53.480695    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg"
Nov 03 00:44:53.495185 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:53.495159    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-0db47e-serving-cert\""
Nov 03 00:44:53.511687 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:53.511654    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg"]
Nov 03 00:44:53.590318 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:53.590286    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg\" (UID: \"c14b1d5d-e75b-45da-a948-9b0a9b7f855e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg"
Nov 03 00:44:53.590318 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:53.590321    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gkv4\" (UniqueName: \"kubernetes.io/projected/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-kube-api-access-2gkv4\") pod \"maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg\" (UID: \"c14b1d5d-e75b-45da-a948-9b0a9b7f855e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg"
Nov 03 00:44:53.590525 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:53.590422    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg\" (UID: \"c14b1d5d-e75b-45da-a948-9b0a9b7f855e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg"
Nov 03 00:44:53.590578 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:53.590519    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-metrics-certs\") pod \"maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg\" (UID: \"c14b1d5d-e75b-45da-a948-9b0a9b7f855e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg"
Nov 03 00:44:53.692018 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:53.691990    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-metrics-certs\") pod \"maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg\" (UID: \"c14b1d5d-e75b-45da-a948-9b0a9b7f855e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg"
Nov 03 00:44:53.692149 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:53.692133    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg\" (UID: \"c14b1d5d-e75b-45da-a948-9b0a9b7f855e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg"
Nov 03 00:44:53.692195 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:53.692161    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gkv4\" (UniqueName: \"kubernetes.io/projected/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-kube-api-access-2gkv4\") pod \"maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg\" (UID: \"c14b1d5d-e75b-45da-a948-9b0a9b7f855e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg"
Nov 03 00:44:53.692263 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:53.692247    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg\" (UID: \"c14b1d5d-e75b-45da-a948-9b0a9b7f855e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg"
Nov 03 00:44:53.692883 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:53.692862    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg\" (UID: \"c14b1d5d-e75b-45da-a948-9b0a9b7f855e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg"
Nov 03 00:44:53.694425 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:53.694401    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-metrics-certs\") pod \"maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg\" (UID: \"c14b1d5d-e75b-45da-a948-9b0a9b7f855e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg"
Nov 03 00:44:53.694533 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:53.694481    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg\" (UID: \"c14b1d5d-e75b-45da-a948-9b0a9b7f855e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg"
Nov 03 00:44:53.710245 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:53.710221    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gkv4\" (UniqueName: \"kubernetes.io/projected/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-kube-api-access-2gkv4\") pod \"maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg\" (UID: \"c14b1d5d-e75b-45da-a948-9b0a9b7f855e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg"
Nov 03 00:44:53.796239 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:53.795382    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg"
Nov 03 00:44:54.114737 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:44:54.114708    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc14b1d5d_e75b_45da_a948_9b0a9b7f855e.slice/crio-0ae645c02a49f2741505c7c897ed5619696cdd8124d944c7159734fc17f73fe3 WatchSource:0}: Error finding container 0ae645c02a49f2741505c7c897ed5619696cdd8124d944c7159734fc17f73fe3: Status 404 returned error can't find the container with id 0ae645c02a49f2741505c7c897ed5619696cdd8124d944c7159734fc17f73fe3
Nov 03 00:44:54.122684 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:54.122659    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg"]
Nov 03 00:44:54.191834 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:54.191801    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg" event={"ID":"c14b1d5d-e75b-45da-a948-9b0a9b7f855e","Type":"ContainerStarted","Data":"0ae645c02a49f2741505c7c897ed5619696cdd8124d944c7159734fc17f73fe3"}
Nov 03 00:44:55.201186 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:55.201146    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg" event={"ID":"c14b1d5d-e75b-45da-a948-9b0a9b7f855e","Type":"ContainerStarted","Data":"1b3c1a46dea803e1b6f37488504e75e6442a1e3647785206d8da6910523a5231"}
Nov 03 00:44:55.201680 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:55.201209    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg"
Nov 03 00:44:55.241136 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:44:55.241089    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg" podStartSLOduration=1.834938104 podStartE2EDuration="2.241074825s" totalImagesPullingTime="406.136721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:55.236617699 +0000 UTC m=+2378.116201745" watchObservedRunningTime="2026-11-03 00:44:55.241074825 +0000 UTC m=+2378.120658868"
Nov 03 00:45:00.259949 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.259741    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t"]
Nov 03 00:45:00.265007 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.264979    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t"
Nov 03 00:45:00.295808 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.295774    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t"]
Nov 03 00:45:00.300510 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.300479    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:45:00.312059 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.312031    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894445-nhkrc"]
Nov 03 00:45:00.320533 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.320511    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l"]
Nov 03 00:45:00.320647 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.320629    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-nhkrc"
Nov 03 00:45:00.325172 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.325152    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l"
Nov 03 00:45:00.348456 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.347978    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894445-nhkrc"]
Nov 03 00:45:00.350209 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.350178    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk2dr\" (UniqueName: \"kubernetes.io/projected/f9128726-ac5c-4073-9525-ed7beca6a72f-kube-api-access-lk2dr\") pod \"maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t\" (UID: \"f9128726-ac5c-4073-9525-ed7beca6a72f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t"
Nov 03 00:45:00.350322 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.350271    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e14c921-01de-45e4-8d0c-65579eea6e32-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l\" (UID: \"6e14c921-01de-45e4-8d0c-65579eea6e32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l"
Nov 03 00:45:00.350443 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.350426    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a5c578e-8a75-47c4-9b00-1e5e4c030488-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894445-nhkrc\" (UID: \"8a5c578e-8a75-47c4-9b00-1e5e4c030488\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-nhkrc"
Nov 03 00:45:00.350489 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.350468    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9128726-ac5c-4073-9525-ed7beca6a72f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t\" (UID: \"f9128726-ac5c-4073-9525-ed7beca6a72f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t"
Nov 03 00:45:00.350552 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.350536    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cspqq\" (UniqueName: \"kubernetes.io/projected/8a5c578e-8a75-47c4-9b00-1e5e4c030488-kube-api-access-cspqq\") pod \"maas-api-key-cleanup-29894445-nhkrc\" (UID: \"8a5c578e-8a75-47c4-9b00-1e5e4c030488\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-nhkrc"
Nov 03 00:45:00.350606 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.350591    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrfwh\" (UniqueName: \"kubernetes.io/projected/6e14c921-01de-45e4-8d0c-65579eea6e32-kube-api-access-hrfwh\") pod \"maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l\" (UID: \"6e14c921-01de-45e4-8d0c-65579eea6e32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l"
Nov 03 00:45:00.353826 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.353803    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x"]
Nov 03 00:45:00.363135 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.360971    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x"
Nov 03 00:45:00.395351 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.381729    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x"]
Nov 03 00:45:00.395351 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.394201    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l"]
Nov 03 00:45:00.458681 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.458634    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb5dj\" (UniqueName: \"kubernetes.io/projected/e8cfbea4-b52d-4a09-b58d-f7f74163caf8-kube-api-access-qb5dj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x\" (UID: \"e8cfbea4-b52d-4a09-b58d-f7f74163caf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x"
Nov 03 00:45:00.458874 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.458757    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk2dr\" (UniqueName: \"kubernetes.io/projected/f9128726-ac5c-4073-9525-ed7beca6a72f-kube-api-access-lk2dr\") pod \"maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t\" (UID: \"f9128726-ac5c-4073-9525-ed7beca6a72f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t"
Nov 03 00:45:00.458874 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.458816    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8cfbea4-b52d-4a09-b58d-f7f74163caf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x\" (UID: \"e8cfbea4-b52d-4a09-b58d-f7f74163caf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x"
Nov 03 00:45:00.458874 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.458855    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e14c921-01de-45e4-8d0c-65579eea6e32-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l\" (UID: \"6e14c921-01de-45e4-8d0c-65579eea6e32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l"
Nov 03 00:45:00.459116 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.459013    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a5c578e-8a75-47c4-9b00-1e5e4c030488-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894445-nhkrc\" (UID: \"8a5c578e-8a75-47c4-9b00-1e5e4c030488\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-nhkrc"
Nov 03 00:45:00.459116 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.459049    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9128726-ac5c-4073-9525-ed7beca6a72f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t\" (UID: \"f9128726-ac5c-4073-9525-ed7beca6a72f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t"
Nov 03 00:45:00.459116 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.459109    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cspqq\" (UniqueName: \"kubernetes.io/projected/8a5c578e-8a75-47c4-9b00-1e5e4c030488-kube-api-access-cspqq\") pod \"maas-api-key-cleanup-29894445-nhkrc\" (UID: \"8a5c578e-8a75-47c4-9b00-1e5e4c030488\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-nhkrc"
Nov 03 00:45:00.459280 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.459158    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrfwh\" (UniqueName: \"kubernetes.io/projected/6e14c921-01de-45e4-8d0c-65579eea6e32-kube-api-access-hrfwh\") pod \"maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l\" (UID: \"6e14c921-01de-45e4-8d0c-65579eea6e32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l"
Nov 03 00:45:00.459738 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.459706    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e14c921-01de-45e4-8d0c-65579eea6e32-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l\" (UID: \"6e14c921-01de-45e4-8d0c-65579eea6e32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l"
Nov 03 00:45:00.460112 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.459757    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9128726-ac5c-4073-9525-ed7beca6a72f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t\" (UID: \"f9128726-ac5c-4073-9525-ed7beca6a72f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t"
Nov 03 00:45:00.460112 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.459835    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a5c578e-8a75-47c4-9b00-1e5e4c030488-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894445-nhkrc\" (UID: \"8a5c578e-8a75-47c4-9b00-1e5e4c030488\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-nhkrc"
Nov 03 00:45:00.476731 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.476704    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrfwh\" (UniqueName: \"kubernetes.io/projected/6e14c921-01de-45e4-8d0c-65579eea6e32-kube-api-access-hrfwh\") pod \"maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l\" (UID: \"6e14c921-01de-45e4-8d0c-65579eea6e32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l"
Nov 03 00:45:00.476986 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.476942    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cspqq\" (UniqueName: \"kubernetes.io/projected/8a5c578e-8a75-47c4-9b00-1e5e4c030488-kube-api-access-cspqq\") pod \"maas-api-key-cleanup-29894445-nhkrc\" (UID: \"8a5c578e-8a75-47c4-9b00-1e5e4c030488\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-nhkrc"
Nov 03 00:45:00.477222 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.477205    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk2dr\" (UniqueName: \"kubernetes.io/projected/f9128726-ac5c-4073-9525-ed7beca6a72f-kube-api-access-lk2dr\") pod \"maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t\" (UID: \"f9128726-ac5c-4073-9525-ed7beca6a72f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t"
Nov 03 00:45:00.560781 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.560695    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb5dj\" (UniqueName: \"kubernetes.io/projected/e8cfbea4-b52d-4a09-b58d-f7f74163caf8-kube-api-access-qb5dj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x\" (UID: \"e8cfbea4-b52d-4a09-b58d-f7f74163caf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x"
Nov 03 00:45:00.560781 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.560775    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8cfbea4-b52d-4a09-b58d-f7f74163caf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x\" (UID: \"e8cfbea4-b52d-4a09-b58d-f7f74163caf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x"
Nov 03 00:45:00.561430 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.561406    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8cfbea4-b52d-4a09-b58d-f7f74163caf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x\" (UID: \"e8cfbea4-b52d-4a09-b58d-f7f74163caf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x"
Nov 03 00:45:00.580655 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.580629    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb5dj\" (UniqueName: \"kubernetes.io/projected/e8cfbea4-b52d-4a09-b58d-f7f74163caf8-kube-api-access-qb5dj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x\" (UID: \"e8cfbea4-b52d-4a09-b58d-f7f74163caf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x"
Nov 03 00:45:00.584423 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.584398    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t"
Nov 03 00:45:00.659090 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.659056    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-nhkrc"
Nov 03 00:45:00.667573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.667320    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l"
Nov 03 00:45:00.691064 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.690800    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x"
Nov 03 00:45:00.902778 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:00.902752    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t"]
Nov 03 00:45:00.903345 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:45:00.903319    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9128726_ac5c_4073_9525_ed7beca6a72f.slice/crio-9fc922c96639205d1216869d73f3d6208f0e5c2a6858699608e44d95e3db1a89 WatchSource:0}: Error finding container 9fc922c96639205d1216869d73f3d6208f0e5c2a6858699608e44d95e3db1a89: Status 404 returned error can't find the container with id 9fc922c96639205d1216869d73f3d6208f0e5c2a6858699608e44d95e3db1a89
Nov 03 00:45:01.032936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:01.032882    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894445-nhkrc"]
Nov 03 00:45:01.039851 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:01.039827    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l"]
Nov 03 00:45:01.089038 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:01.089009    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x"]
Nov 03 00:45:01.090261 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:45:01.090232    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5c578e_8a75_47c4_9b00_1e5e4c030488.slice/crio-a275553f17e7f590e760d532137e339be8f4fc1c8cee0ea6466ade5f96a61a19 WatchSource:0}: Error finding container a275553f17e7f590e760d532137e339be8f4fc1c8cee0ea6466ade5f96a61a19: Status 404 returned error can't find the container with id a275553f17e7f590e760d532137e339be8f4fc1c8cee0ea6466ade5f96a61a19
Nov 03 00:45:01.092947 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:45:01.092889    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e14c921_01de_45e4_8d0c_65579eea6e32.slice/crio-6311f70b5bece6f1766bfb9d56b095aa4202696cdaa5a26334d5c8098e0851c9 WatchSource:0}: Error finding container 6311f70b5bece6f1766bfb9d56b095aa4202696cdaa5a26334d5c8098e0851c9: Status 404 returned error can't find the container with id 6311f70b5bece6f1766bfb9d56b095aa4202696cdaa5a26334d5c8098e0851c9
Nov 03 00:45:01.214303 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:01.214275    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg"
Nov 03 00:45:01.257723 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:01.257687    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l" event={"ID":"6e14c921-01de-45e4-8d0c-65579eea6e32","Type":"ContainerStarted","Data":"6311f70b5bece6f1766bfb9d56b095aa4202696cdaa5a26334d5c8098e0851c9"}
Nov 03 00:45:01.259354 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:01.259324    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x" event={"ID":"e8cfbea4-b52d-4a09-b58d-f7f74163caf8","Type":"ContainerStarted","Data":"1539d1f40497856e14a320254cd89cfc57920e38e09ddef98c9a9d86923a9309"}
Nov 03 00:45:01.261424 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:01.261106    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t" event={"ID":"f9128726-ac5c-4073-9525-ed7beca6a72f","Type":"ContainerStarted","Data":"9fc922c96639205d1216869d73f3d6208f0e5c2a6858699608e44d95e3db1a89"}
Nov 03 00:45:01.262963 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:01.262942    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-nhkrc" event={"ID":"8a5c578e-8a75-47c4-9b00-1e5e4c030488","Type":"ContainerStarted","Data":"a275553f17e7f590e760d532137e339be8f4fc1c8cee0ea6466ade5f96a61a19"}
Nov 03 00:45:02.273029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:02.272998    5012 generic.go:356] "Generic (PLEG): container finished" podID="6e14c921-01de-45e4-8d0c-65579eea6e32" containerID="b376911f0d8169b567ac306faf46a50c8ca6a4d182f97e9a36d180f18f40ad3d" exitCode=60
Nov 03 00:45:02.273414 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:02.273070    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l" event={"ID":"6e14c921-01de-45e4-8d0c-65579eea6e32","Type":"ContainerDied","Data":"b376911f0d8169b567ac306faf46a50c8ca6a4d182f97e9a36d180f18f40ad3d"}
Nov 03 00:45:02.273414 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:02.273293    5012 scope.go:122] "RemoveContainer" containerID="b376911f0d8169b567ac306faf46a50c8ca6a4d182f97e9a36d180f18f40ad3d"
Nov 03 00:45:02.275116 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:02.275096    5012 generic.go:356] "Generic (PLEG): container finished" podID="e8cfbea4-b52d-4a09-b58d-f7f74163caf8" containerID="7f43972d52b13cbec72df337ff1c855e6fa091ec429fa71c88ec93627d055d0d" exitCode=60
Nov 03 00:45:02.275216 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:02.275201    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x" event={"ID":"e8cfbea4-b52d-4a09-b58d-f7f74163caf8","Type":"ContainerDied","Data":"7f43972d52b13cbec72df337ff1c855e6fa091ec429fa71c88ec93627d055d0d"}
Nov 03 00:45:02.275454 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:02.275440    5012 scope.go:122] "RemoveContainer" containerID="7f43972d52b13cbec72df337ff1c855e6fa091ec429fa71c88ec93627d055d0d"
Nov 03 00:45:02.278440 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:02.278418    5012 generic.go:356] "Generic (PLEG): container finished" podID="f9128726-ac5c-4073-9525-ed7beca6a72f" containerID="0b1991a71e29c53dd49a50d277d405dc2a3494e7763e5a61391e69b8e41c1cf9" exitCode=60
Nov 03 00:45:02.278551 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:02.278504    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t" event={"ID":"f9128726-ac5c-4073-9525-ed7beca6a72f","Type":"ContainerDied","Data":"0b1991a71e29c53dd49a50d277d405dc2a3494e7763e5a61391e69b8e41c1cf9"}
Nov 03 00:45:02.278769 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:02.278752    5012 scope.go:122] "RemoveContainer" containerID="0b1991a71e29c53dd49a50d277d405dc2a3494e7763e5a61391e69b8e41c1cf9"
Nov 03 00:45:02.280426 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:02.280399    5012 generic.go:356] "Generic (PLEG): container finished" podID="8a5c578e-8a75-47c4-9b00-1e5e4c030488" containerID="4fae5457b8085663b15f40ff3886d41fe78f0e688b758115d571b5194c62e584" exitCode=60
Nov 03 00:45:02.280550 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:02.280484    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-nhkrc" event={"ID":"8a5c578e-8a75-47c4-9b00-1e5e4c030488","Type":"ContainerDied","Data":"4fae5457b8085663b15f40ff3886d41fe78f0e688b758115d571b5194c62e584"}
Nov 03 00:45:02.280622 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:02.280609    5012 scope.go:122] "RemoveContainer" containerID="4fae5457b8085663b15f40ff3886d41fe78f0e688b758115d571b5194c62e584"
Nov 03 00:45:03.290742 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:03.290708    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l" event={"ID":"6e14c921-01de-45e4-8d0c-65579eea6e32","Type":"ContainerStarted","Data":"744d8df806a366e32a81b2652429e2b9778a5f9bdc223fec8184b4aa3a48637b"}
Nov 03 00:45:03.292869 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:03.292837    5012 generic.go:356] "Generic (PLEG): container finished" podID="e8cfbea4-b52d-4a09-b58d-f7f74163caf8" containerID="f877c498590cfff18676bfac77fdcf72e5f6c6af3e07df860f0aa19c2d8891be" exitCode=60
Nov 03 00:45:03.293003 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:03.292923    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x" event={"ID":"e8cfbea4-b52d-4a09-b58d-f7f74163caf8","Type":"ContainerDied","Data":"f877c498590cfff18676bfac77fdcf72e5f6c6af3e07df860f0aa19c2d8891be"}
Nov 03 00:45:03.293003 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:03.292976    5012 scope.go:122] "RemoveContainer" containerID="7f43972d52b13cbec72df337ff1c855e6fa091ec429fa71c88ec93627d055d0d"
Nov 03 00:45:03.293295 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:03.293270    5012 scope.go:122] "RemoveContainer" containerID="f877c498590cfff18676bfac77fdcf72e5f6c6af3e07df860f0aa19c2d8891be"
Nov 03 00:45:03.293569 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:45:03.293543    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x_odh-ai-gateway-infra(e8cfbea4-b52d-4a09-b58d-f7f74163caf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x" podUID="e8cfbea4-b52d-4a09-b58d-f7f74163caf8"
Nov 03 00:45:03.295097 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:03.295078    5012 generic.go:356] "Generic (PLEG): container finished" podID="f9128726-ac5c-4073-9525-ed7beca6a72f" containerID="6e2abddf6f7cb79ee7396a1e6e12a121e20602eedac1dee1628df76d545cf6a7" exitCode=60
Nov 03 00:45:03.295197 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:03.295126    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t" event={"ID":"f9128726-ac5c-4073-9525-ed7beca6a72f","Type":"ContainerDied","Data":"6e2abddf6f7cb79ee7396a1e6e12a121e20602eedac1dee1628df76d545cf6a7"}
Nov 03 00:45:03.295400 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:03.295379    5012 scope.go:122] "RemoveContainer" containerID="6e2abddf6f7cb79ee7396a1e6e12a121e20602eedac1dee1628df76d545cf6a7"
Nov 03 00:45:03.295668 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:45:03.295645    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t_odh-ai-gateway-infra(f9128726-ac5c-4073-9525-ed7beca6a72f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t" podUID="f9128726-ac5c-4073-9525-ed7beca6a72f"
Nov 03 00:45:03.297243 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:03.297218    5012 generic.go:356] "Generic (PLEG): container finished" podID="8a5c578e-8a75-47c4-9b00-1e5e4c030488" containerID="8a12df4eb82f9145f782f942d8ea662e89b3583806d34db17a18ae098ad6b588" exitCode=60
Nov 03 00:45:03.297350 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:03.297280    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-nhkrc" event={"ID":"8a5c578e-8a75-47c4-9b00-1e5e4c030488","Type":"ContainerDied","Data":"8a12df4eb82f9145f782f942d8ea662e89b3583806d34db17a18ae098ad6b588"}
Nov 03 00:45:03.297530 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:03.297510    5012 scope.go:122] "RemoveContainer" containerID="8a12df4eb82f9145f782f942d8ea662e89b3583806d34db17a18ae098ad6b588"
Nov 03 00:45:03.297726 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:45:03.297708    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894445-nhkrc_odh-ai-gateway-infra(8a5c578e-8a75-47c4-9b00-1e5e4c030488)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-nhkrc" podUID="8a5c578e-8a75-47c4-9b00-1e5e4c030488"
Nov 03 00:45:03.331421 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:03.331370    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l" podStartSLOduration=3.331354037 podStartE2EDuration="3.331354037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:03.327443283 +0000 UTC m=+2386.207027340" watchObservedRunningTime="2026-11-03 00:45:03.331354037 +0000 UTC m=+2386.210938084"
Nov 03 00:45:03.491763 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:03.491738    5012 scope.go:122] "RemoveContainer" containerID="0b1991a71e29c53dd49a50d277d405dc2a3494e7763e5a61391e69b8e41c1cf9"
Nov 03 00:45:03.505627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:03.505591    5012 scope.go:122] "RemoveContainer" containerID="4fae5457b8085663b15f40ff3886d41fe78f0e688b758115d571b5194c62e584"
Nov 03 00:45:04.308871 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:04.308839    5012 scope.go:122] "RemoveContainer" containerID="8a12df4eb82f9145f782f942d8ea662e89b3583806d34db17a18ae098ad6b588"
Nov 03 00:45:04.309371 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:45:04.309252    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894445-nhkrc_odh-ai-gateway-infra(8a5c578e-8a75-47c4-9b00-1e5e4c030488)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-nhkrc" podUID="8a5c578e-8a75-47c4-9b00-1e5e4c030488"
Nov 03 00:45:04.310849 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:04.310820    5012 generic.go:356] "Generic (PLEG): container finished" podID="6e14c921-01de-45e4-8d0c-65579eea6e32" containerID="744d8df806a366e32a81b2652429e2b9778a5f9bdc223fec8184b4aa3a48637b" exitCode=60
Nov 03 00:45:04.311018 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:04.310858    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l" event={"ID":"6e14c921-01de-45e4-8d0c-65579eea6e32","Type":"ContainerDied","Data":"744d8df806a366e32a81b2652429e2b9778a5f9bdc223fec8184b4aa3a48637b"}
Nov 03 00:45:04.311018 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:04.310953    5012 scope.go:122] "RemoveContainer" containerID="b376911f0d8169b567ac306faf46a50c8ca6a4d182f97e9a36d180f18f40ad3d"
Nov 03 00:45:04.311256 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:04.311237    5012 scope.go:122] "RemoveContainer" containerID="744d8df806a366e32a81b2652429e2b9778a5f9bdc223fec8184b4aa3a48637b"
Nov 03 00:45:04.311617 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:45:04.311580    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l_odh-ai-gateway-infra(6e14c921-01de-45e4-8d0c-65579eea6e32)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l" podUID="6e14c921-01de-45e4-8d0c-65579eea6e32"
Nov 03 00:45:04.313213 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:04.313190    5012 scope.go:122] "RemoveContainer" containerID="f877c498590cfff18676bfac77fdcf72e5f6c6af3e07df860f0aa19c2d8891be"
Nov 03 00:45:04.313529 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:45:04.313498    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x_odh-ai-gateway-infra(e8cfbea4-b52d-4a09-b58d-f7f74163caf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x" podUID="e8cfbea4-b52d-4a09-b58d-f7f74163caf8"
Nov 03 00:45:04.314800 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:04.314782    5012 scope.go:122] "RemoveContainer" containerID="6e2abddf6f7cb79ee7396a1e6e12a121e20602eedac1dee1628df76d545cf6a7"
Nov 03 00:45:04.315074 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:45:04.315055    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t_odh-ai-gateway-infra(f9128726-ac5c-4073-9525-ed7beca6a72f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t" podUID="f9128726-ac5c-4073-9525-ed7beca6a72f"
Nov 03 00:45:05.324347 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:05.324314    5012 scope.go:122] "RemoveContainer" containerID="744d8df806a366e32a81b2652429e2b9778a5f9bdc223fec8184b4aa3a48637b"
Nov 03 00:45:05.324753 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:45:05.324577    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l_odh-ai-gateway-infra(6e14c921-01de-45e4-8d0c-65579eea6e32)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l" podUID="6e14c921-01de-45e4-8d0c-65579eea6e32"
Nov 03 00:45:07.366171 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:45:07.366132    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:45:14.240934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:14.238192    5012 scope.go:122] "RemoveContainer" containerID="744d8df806a366e32a81b2652429e2b9778a5f9bdc223fec8184b4aa3a48637b"
Nov 03 00:45:14.240934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:14.239186    5012 scope.go:122] "RemoveContainer" containerID="f877c498590cfff18676bfac77fdcf72e5f6c6af3e07df860f0aa19c2d8891be"
Nov 03 00:45:15.429116 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:15.429016    5012 generic.go:356] "Generic (PLEG): container finished" podID="6e14c921-01de-45e4-8d0c-65579eea6e32" containerID="5f8c5dabb9818d18539b15a4c66e6cbcd815091a3e2d8156feec90ec1aa507ef" exitCode=60
Nov 03 00:45:15.429116 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:15.429091    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l" event={"ID":"6e14c921-01de-45e4-8d0c-65579eea6e32","Type":"ContainerDied","Data":"5f8c5dabb9818d18539b15a4c66e6cbcd815091a3e2d8156feec90ec1aa507ef"}
Nov 03 00:45:15.429679 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:15.429175    5012 scope.go:122] "RemoveContainer" containerID="744d8df806a366e32a81b2652429e2b9778a5f9bdc223fec8184b4aa3a48637b"
Nov 03 00:45:15.429679 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:15.429603    5012 scope.go:122] "RemoveContainer" containerID="5f8c5dabb9818d18539b15a4c66e6cbcd815091a3e2d8156feec90ec1aa507ef"
Nov 03 00:45:15.430096 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:45:15.429968    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l_odh-ai-gateway-infra(6e14c921-01de-45e4-8d0c-65579eea6e32)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l" podUID="6e14c921-01de-45e4-8d0c-65579eea6e32"
Nov 03 00:45:15.431846 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:15.431815    5012 generic.go:356] "Generic (PLEG): container finished" podID="e8cfbea4-b52d-4a09-b58d-f7f74163caf8" containerID="8823b02be32351598e86c17c91ff69639cad07b3a5aebc10990bd67c0bd5c918" exitCode=60
Nov 03 00:45:15.431985 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:15.431865    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x" event={"ID":"e8cfbea4-b52d-4a09-b58d-f7f74163caf8","Type":"ContainerDied","Data":"8823b02be32351598e86c17c91ff69639cad07b3a5aebc10990bd67c0bd5c918"}
Nov 03 00:45:15.432249 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:15.432230    5012 scope.go:122] "RemoveContainer" containerID="8823b02be32351598e86c17c91ff69639cad07b3a5aebc10990bd67c0bd5c918"
Nov 03 00:45:15.432512 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:45:15.432494    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x_odh-ai-gateway-infra(e8cfbea4-b52d-4a09-b58d-f7f74163caf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x" podUID="e8cfbea4-b52d-4a09-b58d-f7f74163caf8"
Nov 03 00:45:15.443852 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:15.443836    5012 scope.go:122] "RemoveContainer" containerID="f877c498590cfff18676bfac77fdcf72e5f6c6af3e07df860f0aa19c2d8891be"
Nov 03 00:45:16.236565 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:16.236529    5012 scope.go:122] "RemoveContainer" containerID="6e2abddf6f7cb79ee7396a1e6e12a121e20602eedac1dee1628df76d545cf6a7"
Nov 03 00:45:16.236774 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:16.236650    5012 scope.go:122] "RemoveContainer" containerID="8a12df4eb82f9145f782f942d8ea662e89b3583806d34db17a18ae098ad6b588"
Nov 03 00:45:16.478096 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:16.478067    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l"]
Nov 03 00:45:16.516314 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:16.516280    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x"]
Nov 03 00:45:16.727135 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:16.727105    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l"
Nov 03 00:45:16.847852 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:16.847816    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e14c921-01de-45e4-8d0c-65579eea6e32-kube-api-access-hrfwh\" (UniqueName: \"kubernetes.io/projected/6e14c921-01de-45e4-8d0c-65579eea6e32-kube-api-access-hrfwh\") pod \"6e14c921-01de-45e4-8d0c-65579eea6e32\" (UID: \"6e14c921-01de-45e4-8d0c-65579eea6e32\") "
Nov 03 00:45:16.848030 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:16.847979    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e14c921-01de-45e4-8d0c-65579eea6e32-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e14c921-01de-45e4-8d0c-65579eea6e32-maas-api-service-ca\") pod \"6e14c921-01de-45e4-8d0c-65579eea6e32\" (UID: \"6e14c921-01de-45e4-8d0c-65579eea6e32\") "
Nov 03 00:45:16.848587 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:16.848545    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e14c921-01de-45e4-8d0c-65579eea6e32-maas-api-service-ca" pod "6e14c921-01de-45e4-8d0c-65579eea6e32" (UID: "6e14c921-01de-45e4-8d0c-65579eea6e32"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:16.849424 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:16.849394    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e14c921-01de-45e4-8d0c-65579eea6e32-maas-api-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:45:16.850694 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:16.850668    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e14c921-01de-45e4-8d0c-65579eea6e32-kube-api-access-hrfwh" pod "6e14c921-01de-45e4-8d0c-65579eea6e32" (UID: "6e14c921-01de-45e4-8d0c-65579eea6e32"). InnerVolumeSpecName "kube-api-access-hrfwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:16.856803 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:16.856777    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x"
Nov 03 00:45:16.950033 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:16.949996    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8cfbea4-b52d-4a09-b58d-f7f74163caf8-kube-api-access-qb5dj\" (UniqueName: \"kubernetes.io/projected/e8cfbea4-b52d-4a09-b58d-f7f74163caf8-kube-api-access-qb5dj\") pod \"e8cfbea4-b52d-4a09-b58d-f7f74163caf8\" (UID: \"e8cfbea4-b52d-4a09-b58d-f7f74163caf8\") "
Nov 03 00:45:16.950201 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:16.950124    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8cfbea4-b52d-4a09-b58d-f7f74163caf8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8cfbea4-b52d-4a09-b58d-f7f74163caf8-maas-api-service-ca\") pod \"e8cfbea4-b52d-4a09-b58d-f7f74163caf8\" (UID: \"e8cfbea4-b52d-4a09-b58d-f7f74163caf8\") "
Nov 03 00:45:16.950548 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:16.950515    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8cfbea4-b52d-4a09-b58d-f7f74163caf8-maas-api-service-ca" pod "e8cfbea4-b52d-4a09-b58d-f7f74163caf8" (UID: "e8cfbea4-b52d-4a09-b58d-f7f74163caf8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:16.950700 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:16.950684    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrfwh\" (UniqueName: \"kubernetes.io/projected/6e14c921-01de-45e4-8d0c-65579eea6e32-kube-api-access-hrfwh\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:45:16.950756 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:16.950706    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8cfbea4-b52d-4a09-b58d-f7f74163caf8-maas-api-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:45:16.952030 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:16.952003    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8cfbea4-b52d-4a09-b58d-f7f74163caf8-kube-api-access-qb5dj" pod "e8cfbea4-b52d-4a09-b58d-f7f74163caf8" (UID: "e8cfbea4-b52d-4a09-b58d-f7f74163caf8"). InnerVolumeSpecName "kube-api-access-qb5dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:17.052059 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:17.051896    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb5dj\" (UniqueName: \"kubernetes.io/projected/e8cfbea4-b52d-4a09-b58d-f7f74163caf8-kube-api-access-qb5dj\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:45:17.453744 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:17.453714    5012 generic.go:356] "Generic (PLEG): container finished" podID="8a5c578e-8a75-47c4-9b00-1e5e4c030488" containerID="e7db70abc72668a6cc468f3836f0050c2e1e9d8dcfbf2072ef7cea4d8eb4c614" exitCode=60
Nov 03 00:45:17.453862 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:17.453780    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-nhkrc" event={"ID":"8a5c578e-8a75-47c4-9b00-1e5e4c030488","Type":"ContainerDied","Data":"e7db70abc72668a6cc468f3836f0050c2e1e9d8dcfbf2072ef7cea4d8eb4c614"}
Nov 03 00:45:17.453862 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:17.453839    5012 scope.go:122] "RemoveContainer" containerID="8a12df4eb82f9145f782f942d8ea662e89b3583806d34db17a18ae098ad6b588"
Nov 03 00:45:17.454218 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:17.454194    5012 scope.go:122] "RemoveContainer" containerID="e7db70abc72668a6cc468f3836f0050c2e1e9d8dcfbf2072ef7cea4d8eb4c614"
Nov 03 00:45:17.454546 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:45:17.454525    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894445-nhkrc_odh-ai-gateway-infra(8a5c578e-8a75-47c4-9b00-1e5e4c030488)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-nhkrc" podUID="8a5c578e-8a75-47c4-9b00-1e5e4c030488"
Nov 03 00:45:17.455538 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:17.455516    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l" event={"ID":"6e14c921-01de-45e4-8d0c-65579eea6e32","Type":"ContainerDied","Data":"6311f70b5bece6f1766bfb9d56b095aa4202696cdaa5a26334d5c8098e0851c9"}
Nov 03 00:45:17.455628 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:17.455520    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l"
Nov 03 00:45:17.456891 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:17.456874    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x"
Nov 03 00:45:17.456998 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:17.456879    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x" event={"ID":"e8cfbea4-b52d-4a09-b58d-f7f74163caf8","Type":"ContainerDied","Data":"1539d1f40497856e14a320254cd89cfc57920e38e09ddef98c9a9d86923a9309"}
Nov 03 00:45:17.459549 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:17.459523    5012 generic.go:356] "Generic (PLEG): container finished" podID="f9128726-ac5c-4073-9525-ed7beca6a72f" containerID="4ef185975c334e32910947bd8e6ff016385bb7a0be01b35684404f482fc0efaa" exitCode=60
Nov 03 00:45:17.459659 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:17.459622    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t" event={"ID":"f9128726-ac5c-4073-9525-ed7beca6a72f","Type":"ContainerDied","Data":"4ef185975c334e32910947bd8e6ff016385bb7a0be01b35684404f482fc0efaa"}
Nov 03 00:45:17.460069 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:17.460049    5012 scope.go:122] "RemoveContainer" containerID="4ef185975c334e32910947bd8e6ff016385bb7a0be01b35684404f482fc0efaa"
Nov 03 00:45:17.460344 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:45:17.460320    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t_odh-ai-gateway-infra(f9128726-ac5c-4073-9525-ed7beca6a72f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t" podUID="f9128726-ac5c-4073-9525-ed7beca6a72f"
Nov 03 00:45:17.471485 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:17.471468    5012 scope.go:122] "RemoveContainer" containerID="5f8c5dabb9818d18539b15a4c66e6cbcd815091a3e2d8156feec90ec1aa507ef"
Nov 03 00:45:17.480955 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:17.480746    5012 scope.go:122] "RemoveContainer" containerID="8823b02be32351598e86c17c91ff69639cad07b3a5aebc10990bd67c0bd5c918"
Nov 03 00:45:17.489978 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:17.489962    5012 scope.go:122] "RemoveContainer" containerID="6e2abddf6f7cb79ee7396a1e6e12a121e20602eedac1dee1628df76d545cf6a7"
Nov 03 00:45:17.525760 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:17.525740    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x"]
Nov 03 00:45:17.532317 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:17.532295    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894445-lc99x"]
Nov 03 00:45:17.598681 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:17.598655    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l"]
Nov 03 00:45:17.602826 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:17.602802    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894445-zhd4l"]
Nov 03 00:45:18.157495 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:18.157469    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 00:45:18.161757 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:18.161731    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:45:18.172572 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:18.172554    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 00:45:18.176802 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:18.176784    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:45:18.501681 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:18.501605    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894445-nhkrc"]
Nov 03 00:45:18.590056 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:18.590024    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t"]
Nov 03 00:45:18.685452 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:18.685423    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-nhkrc"
Nov 03 00:45:18.789775 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:18.787061    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5c578e-8a75-47c4-9b00-1e5e4c030488-kube-api-access-cspqq\" (UniqueName: \"kubernetes.io/projected/8a5c578e-8a75-47c4-9b00-1e5e4c030488-kube-api-access-cspqq\") pod \"8a5c578e-8a75-47c4-9b00-1e5e4c030488\" (UID: \"8a5c578e-8a75-47c4-9b00-1e5e4c030488\") "
Nov 03 00:45:18.789775 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:18.787185    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a5c578e-8a75-47c4-9b00-1e5e4c030488-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a5c578e-8a75-47c4-9b00-1e5e4c030488-maas-api-service-ca\") pod \"8a5c578e-8a75-47c4-9b00-1e5e4c030488\" (UID: \"8a5c578e-8a75-47c4-9b00-1e5e4c030488\") "
Nov 03 00:45:18.789775 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:18.788326    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a5c578e-8a75-47c4-9b00-1e5e4c030488-maas-api-service-ca" pod "8a5c578e-8a75-47c4-9b00-1e5e4c030488" (UID: "8a5c578e-8a75-47c4-9b00-1e5e4c030488"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:18.791119 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:18.791089    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5c578e-8a75-47c4-9b00-1e5e4c030488-kube-api-access-cspqq" pod "8a5c578e-8a75-47c4-9b00-1e5e4c030488" (UID: "8a5c578e-8a75-47c4-9b00-1e5e4c030488"). InnerVolumeSpecName "kube-api-access-cspqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:18.843696 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:18.843671    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t"
Nov 03 00:45:18.890729 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:18.890669    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cspqq\" (UniqueName: \"kubernetes.io/projected/8a5c578e-8a75-47c4-9b00-1e5e4c030488-kube-api-access-cspqq\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:45:18.890729 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:18.890702    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a5c578e-8a75-47c4-9b00-1e5e4c030488-maas-api-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:45:18.992043 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:18.992016    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9128726-ac5c-4073-9525-ed7beca6a72f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9128726-ac5c-4073-9525-ed7beca6a72f-maas-api-service-ca\") pod \"f9128726-ac5c-4073-9525-ed7beca6a72f\" (UID: \"f9128726-ac5c-4073-9525-ed7beca6a72f\") "
Nov 03 00:45:18.992207 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:18.992086    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9128726-ac5c-4073-9525-ed7beca6a72f-kube-api-access-lk2dr\" (UniqueName: \"kubernetes.io/projected/f9128726-ac5c-4073-9525-ed7beca6a72f-kube-api-access-lk2dr\") pod \"f9128726-ac5c-4073-9525-ed7beca6a72f\" (UID: \"f9128726-ac5c-4073-9525-ed7beca6a72f\") "
Nov 03 00:45:18.992389 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:18.992353    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9128726-ac5c-4073-9525-ed7beca6a72f-maas-api-service-ca" pod "f9128726-ac5c-4073-9525-ed7beca6a72f" (UID: "f9128726-ac5c-4073-9525-ed7beca6a72f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:18.992595 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:18.992577    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9128726-ac5c-4073-9525-ed7beca6a72f-maas-api-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:45:18.994148 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:18.994128    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9128726-ac5c-4073-9525-ed7beca6a72f-kube-api-access-lk2dr" pod "f9128726-ac5c-4073-9525-ed7beca6a72f" (UID: "f9128726-ac5c-4073-9525-ed7beca6a72f"). InnerVolumeSpecName "kube-api-access-lk2dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:19.093620 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:19.093565    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk2dr\" (UniqueName: \"kubernetes.io/projected/f9128726-ac5c-4073-9525-ed7beca6a72f-kube-api-access-lk2dr\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:45:19.249438 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:19.249403    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e14c921-01de-45e4-8d0c-65579eea6e32" path="/var/lib/kubelet/pods/6e14c921-01de-45e4-8d0c-65579eea6e32/volumes"
Nov 03 00:45:19.250160 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:19.250135    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8cfbea4-b52d-4a09-b58d-f7f74163caf8" path="/var/lib/kubelet/pods/e8cfbea4-b52d-4a09-b58d-f7f74163caf8/volumes"
Nov 03 00:45:19.481656 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:19.481630    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-nhkrc"
Nov 03 00:45:19.481820 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:19.481667    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-nhkrc" event={"ID":"8a5c578e-8a75-47c4-9b00-1e5e4c030488","Type":"ContainerDied","Data":"a275553f17e7f590e760d532137e339be8f4fc1c8cee0ea6466ade5f96a61a19"}
Nov 03 00:45:19.481820 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:19.481723    5012 scope.go:122] "RemoveContainer" containerID="e7db70abc72668a6cc468f3836f0050c2e1e9d8dcfbf2072ef7cea4d8eb4c614"
Nov 03 00:45:19.483532 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:19.483508    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t" event={"ID":"f9128726-ac5c-4073-9525-ed7beca6a72f","Type":"ContainerDied","Data":"9fc922c96639205d1216869d73f3d6208f0e5c2a6858699608e44d95e3db1a89"}
Nov 03 00:45:19.483532 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:19.483528    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t"
Nov 03 00:45:19.492685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:19.492670    5012 scope.go:122] "RemoveContainer" containerID="4ef185975c334e32910947bd8e6ff016385bb7a0be01b35684404f482fc0efaa"
Nov 03 00:45:19.529175 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:19.529153    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894445-nhkrc"]
Nov 03 00:45:19.543038 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:19.543012    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894445-nhkrc"]
Nov 03 00:45:19.568791 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:19.568764    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t"]
Nov 03 00:45:19.572592 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:19.572571    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894445-8qq7t"]
Nov 03 00:45:21.248433 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:21.248396    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a5c578e-8a75-47c4-9b00-1e5e4c030488" path="/var/lib/kubelet/pods/8a5c578e-8a75-47c4-9b00-1e5e4c030488/volumes"
Nov 03 00:45:21.249015 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:21.248995    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9128726-ac5c-4073-9525-ed7beca6a72f" path="/var/lib/kubelet/pods/f9128726-ac5c-4073-9525-ed7beca6a72f/volumes"
Nov 03 00:45:32.400083 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.400046    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7"]
Nov 03 00:45:32.401154 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.401131    5012 cpu_manager.go:414] "containerMap: removing container" podUID="8a5c578e-8a75-47c4-9b00-1e5e4c030488" containerName="cleanup"
Nov 03 00:45:32.401154 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.401155    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5c578e-8a75-47c4-9b00-1e5e4c030488" containerName="cleanup"
Nov 03 00:45:32.401311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.401174    5012 cpu_manager.go:414] "containerMap: removing container" podUID="e8cfbea4-b52d-4a09-b58d-f7f74163caf8" containerName="cleanup"
Nov 03 00:45:32.401311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.401183    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8cfbea4-b52d-4a09-b58d-f7f74163caf8" containerName="cleanup"
Nov 03 00:45:32.401311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.401197    5012 cpu_manager.go:414] "containerMap: removing container" podUID="e8cfbea4-b52d-4a09-b58d-f7f74163caf8" containerName="cleanup"
Nov 03 00:45:32.401311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.401204    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8cfbea4-b52d-4a09-b58d-f7f74163caf8" containerName="cleanup"
Nov 03 00:45:32.401311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.401215    5012 cpu_manager.go:414] "containerMap: removing container" podUID="6e14c921-01de-45e4-8d0c-65579eea6e32" containerName="cleanup"
Nov 03 00:45:32.401311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.401222    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e14c921-01de-45e4-8d0c-65579eea6e32" containerName="cleanup"
Nov 03 00:45:32.401311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.401232    5012 cpu_manager.go:414] "containerMap: removing container" podUID="f9128726-ac5c-4073-9525-ed7beca6a72f" containerName="cleanup"
Nov 03 00:45:32.401311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.401239    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9128726-ac5c-4073-9525-ed7beca6a72f" containerName="cleanup"
Nov 03 00:45:32.401311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.401272    5012 cpu_manager.go:414] "containerMap: removing container" podUID="8a5c578e-8a75-47c4-9b00-1e5e4c030488" containerName="cleanup"
Nov 03 00:45:32.401311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.401280    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5c578e-8a75-47c4-9b00-1e5e4c030488" containerName="cleanup"
Nov 03 00:45:32.401311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.401303    5012 cpu_manager.go:414] "containerMap: removing container" podUID="6e14c921-01de-45e4-8d0c-65579eea6e32" containerName="cleanup"
Nov 03 00:45:32.401311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.401310    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e14c921-01de-45e4-8d0c-65579eea6e32" containerName="cleanup"
Nov 03 00:45:32.402044 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.401325    5012 cpu_manager.go:414] "containerMap: removing container" podUID="f9128726-ac5c-4073-9525-ed7beca6a72f" containerName="cleanup"
Nov 03 00:45:32.402044 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.401331    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9128726-ac5c-4073-9525-ed7beca6a72f" containerName="cleanup"
Nov 03 00:45:32.402044 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.401496    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e14c921-01de-45e4-8d0c-65579eea6e32" containerName="cleanup"
Nov 03 00:45:32.402044 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.401510    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9128726-ac5c-4073-9525-ed7beca6a72f" containerName="cleanup"
Nov 03 00:45:32.402044 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.401521    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5c578e-8a75-47c4-9b00-1e5e4c030488" containerName="cleanup"
Nov 03 00:45:32.402044 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.401536    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5c578e-8a75-47c4-9b00-1e5e4c030488" containerName="cleanup"
Nov 03 00:45:32.402044 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.401545    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8cfbea4-b52d-4a09-b58d-f7f74163caf8" containerName="cleanup"
Nov 03 00:45:32.402044 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.401554    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8cfbea4-b52d-4a09-b58d-f7f74163caf8" containerName="cleanup"
Nov 03 00:45:32.402044 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.401566    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8cfbea4-b52d-4a09-b58d-f7f74163caf8" containerName="cleanup"
Nov 03 00:45:32.402044 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.401576    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e14c921-01de-45e4-8d0c-65579eea6e32" containerName="cleanup"
Nov 03 00:45:32.402044 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.401583    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9128726-ac5c-4073-9525-ed7beca6a72f" containerName="cleanup"
Nov 03 00:45:32.407579 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.407561    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7"
Nov 03 00:45:32.424055 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.424026    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7"]
Nov 03 00:45:32.429748 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.429631    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-ac338d-serving-cert\""
Nov 03 00:45:32.472893 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.472854    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26130d50-27b4-4de9-8ec5-03af2ae23cbc-maas-api-tls\") pod \"maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7\" (UID: \"26130d50-27b4-4de9-8ec5-03af2ae23cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7"
Nov 03 00:45:32.473095 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.473007    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26130d50-27b4-4de9-8ec5-03af2ae23cbc-metrics-certs\") pod \"maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7\" (UID: \"26130d50-27b4-4de9-8ec5-03af2ae23cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7"
Nov 03 00:45:32.473179 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.473160    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26130d50-27b4-4de9-8ec5-03af2ae23cbc-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7\" (UID: \"26130d50-27b4-4de9-8ec5-03af2ae23cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7"
Nov 03 00:45:32.473243 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.473188    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrlr8\" (UniqueName: \"kubernetes.io/projected/26130d50-27b4-4de9-8ec5-03af2ae23cbc-kube-api-access-lrlr8\") pod \"maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7\" (UID: \"26130d50-27b4-4de9-8ec5-03af2ae23cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7"
Nov 03 00:45:32.574418 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.574378    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26130d50-27b4-4de9-8ec5-03af2ae23cbc-maas-api-tls\") pod \"maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7\" (UID: \"26130d50-27b4-4de9-8ec5-03af2ae23cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7"
Nov 03 00:45:32.574627 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.574497    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26130d50-27b4-4de9-8ec5-03af2ae23cbc-metrics-certs\") pod \"maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7\" (UID: \"26130d50-27b4-4de9-8ec5-03af2ae23cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7"
Nov 03 00:45:32.574713 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.574646    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26130d50-27b4-4de9-8ec5-03af2ae23cbc-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7\" (UID: \"26130d50-27b4-4de9-8ec5-03af2ae23cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7"
Nov 03 00:45:32.574713 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.574689    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrlr8\" (UniqueName: \"kubernetes.io/projected/26130d50-27b4-4de9-8ec5-03af2ae23cbc-kube-api-access-lrlr8\") pod \"maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7\" (UID: \"26130d50-27b4-4de9-8ec5-03af2ae23cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7"
Nov 03 00:45:32.575532 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.575500    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26130d50-27b4-4de9-8ec5-03af2ae23cbc-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7\" (UID: \"26130d50-27b4-4de9-8ec5-03af2ae23cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7"
Nov 03 00:45:32.577194 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.577170    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26130d50-27b4-4de9-8ec5-03af2ae23cbc-metrics-certs\") pod \"maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7\" (UID: \"26130d50-27b4-4de9-8ec5-03af2ae23cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7"
Nov 03 00:45:32.577310 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.577192    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26130d50-27b4-4de9-8ec5-03af2ae23cbc-maas-api-tls\") pod \"maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7\" (UID: \"26130d50-27b4-4de9-8ec5-03af2ae23cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7"
Nov 03 00:45:32.591982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.591960    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrlr8\" (UniqueName: \"kubernetes.io/projected/26130d50-27b4-4de9-8ec5-03af2ae23cbc-kube-api-access-lrlr8\") pod \"maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7\" (UID: \"26130d50-27b4-4de9-8ec5-03af2ae23cbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7"
Nov 03 00:45:32.740979 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:32.740141    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7"
Nov 03 00:45:33.256574 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:45:33.256491    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26130d50_27b4_4de9_8ec5_03af2ae23cbc.slice/crio-d9f1def129ac9cb565a0c103998d59b118c4ebfa5d6d0c5ffbabdc27e535c9bc WatchSource:0}: Error finding container d9f1def129ac9cb565a0c103998d59b118c4ebfa5d6d0c5ffbabdc27e535c9bc: Status 404 returned error can't find the container with id d9f1def129ac9cb565a0c103998d59b118c4ebfa5d6d0c5ffbabdc27e535c9bc
Nov 03 00:45:33.259949 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:33.259218    5012 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:45:33.273507 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:33.273472    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7"]
Nov 03 00:45:33.607818 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:33.607784    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7" event={"ID":"26130d50-27b4-4de9-8ec5-03af2ae23cbc","Type":"ContainerStarted","Data":"d9f1def129ac9cb565a0c103998d59b118c4ebfa5d6d0c5ffbabdc27e535c9bc"}
Nov 03 00:45:34.617960 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:34.617881    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7" event={"ID":"26130d50-27b4-4de9-8ec5-03af2ae23cbc","Type":"ContainerStarted","Data":"96a5ba0efcca8045432e5c12080bff3cc5e22c3767fb13d90654ae388a450d06"}
Nov 03 00:45:34.618384 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:34.618001    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7"
Nov 03 00:45:34.655175 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:34.655122    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7" podStartSLOduration=2.230004993 podStartE2EDuration="2.65510866s" totalImagesPullingTime="425.103667ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:34.652629826 +0000 UTC m=+2417.532213869" watchObservedRunningTime="2026-11-03 00:45:34.65510866 +0000 UTC m=+2417.534692703"
Nov 03 00:45:37.362905 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:45:37.362852    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:45:40.631806 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:45:40.631770    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7"
Nov 03 00:46:07.363473 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:46:07.363436    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:46:08.017343 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:08.017307    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"]
Nov 03 00:46:08.017596 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:08.017559    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9" podUID="8a090865-de87-442c-806a-76e91d5d374c" containerName="istio-proxy" containerID="cri-o://dd014bbb64d8d2bff6703fbf3b4e5022dd27c5703717c686e1c67310e75023ae" gracePeriod=30
Nov 03 00:46:13.509409 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.509380    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:46:13.579382 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.579351    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a090865-de87-442c-806a-76e91d5d374c-kube-api-access-v252f\" (UniqueName: \"kubernetes.io/projected/8a090865-de87-442c-806a-76e91d5d374c-kube-api-access-v252f\") pod \"8a090865-de87-442c-806a-76e91d5d374c\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") "
Nov 03 00:46:13.579569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.579429    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-istio-envoy\") pod \"8a090865-de87-442c-806a-76e91d5d374c\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") "
Nov 03 00:46:13.579569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.579472    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-workload-certs\") pod \"8a090865-de87-442c-806a-76e91d5d374c\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") "
Nov 03 00:46:13.579569 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.579548    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a090865-de87-442c-806a-76e91d5d374c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a090865-de87-442c-806a-76e91d5d374c-istiod-ca-cert\") pod \"8a090865-de87-442c-806a-76e91d5d374c\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") "
Nov 03 00:46:13.579764 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.579580    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-workload-socket\") pod \"8a090865-de87-442c-806a-76e91d5d374c\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") "
Nov 03 00:46:13.579764 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.579601    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-istio-data\") pod \"8a090865-de87-442c-806a-76e91d5d374c\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") "
Nov 03 00:46:13.579764 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.579621    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a090865-de87-442c-806a-76e91d5d374c-istio-token\" (UniqueName: \"kubernetes.io/projected/8a090865-de87-442c-806a-76e91d5d374c-istio-token\") pod \"8a090865-de87-442c-806a-76e91d5d374c\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") "
Nov 03 00:46:13.579764 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.579670    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a090865-de87-442c-806a-76e91d5d374c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a090865-de87-442c-806a-76e91d5d374c-istio-podinfo\") pod \"8a090865-de87-442c-806a-76e91d5d374c\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") "
Nov 03 00:46:13.579764 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.579715    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-credential-socket\") pod \"8a090865-de87-442c-806a-76e91d5d374c\" (UID: \"8a090865-de87-442c-806a-76e91d5d374c\") "
Nov 03 00:46:13.580091 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.580020    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-workload-socket" pod "8a090865-de87-442c-806a-76e91d5d374c" (UID: "8a090865-de87-442c-806a-76e91d5d374c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:46:13.580375 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.580350    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:46:13.580693 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.580667    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-workload-certs" pod "8a090865-de87-442c-806a-76e91d5d374c" (UID: "8a090865-de87-442c-806a-76e91d5d374c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:46:13.581035 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.581012    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-credential-socket" pod "8a090865-de87-442c-806a-76e91d5d374c" (UID: "8a090865-de87-442c-806a-76e91d5d374c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:46:13.581308 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.581268    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a090865-de87-442c-806a-76e91d5d374c-istiod-ca-cert" pod "8a090865-de87-442c-806a-76e91d5d374c" (UID: "8a090865-de87-442c-806a-76e91d5d374c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:46:13.581513 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.581468    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-istio-data" pod "8a090865-de87-442c-806a-76e91d5d374c" (UID: "8a090865-de87-442c-806a-76e91d5d374c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:46:13.582496 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.582470    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a090865-de87-442c-806a-76e91d5d374c-kube-api-access-v252f" pod "8a090865-de87-442c-806a-76e91d5d374c" (UID: "8a090865-de87-442c-806a-76e91d5d374c"). InnerVolumeSpecName "kube-api-access-v252f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:46:13.582496 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.582482    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-istio-envoy" pod "8a090865-de87-442c-806a-76e91d5d374c" (UID: "8a090865-de87-442c-806a-76e91d5d374c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:46:13.582783 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.582751    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a090865-de87-442c-806a-76e91d5d374c-istio-token" pod "8a090865-de87-442c-806a-76e91d5d374c" (UID: "8a090865-de87-442c-806a-76e91d5d374c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:46:13.583109 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.583089    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a090865-de87-442c-806a-76e91d5d374c-istio-podinfo" pod "8a090865-de87-442c-806a-76e91d5d374c" (UID: "8a090865-de87-442c-806a-76e91d5d374c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:46:13.681556 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.681520    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a090865-de87-442c-806a-76e91d5d374c-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:46:13.681556 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.681555    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:46:13.681771 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.681566    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a090865-de87-442c-806a-76e91d5d374c-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:46:13.681771 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.681578    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a090865-de87-442c-806a-76e91d5d374c-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:46:13.681771 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.681589    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:46:13.681771 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.681602    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v252f\" (UniqueName: \"kubernetes.io/projected/8a090865-de87-442c-806a-76e91d5d374c-kube-api-access-v252f\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:46:13.681771 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.681613    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:46:13.681771 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.681626    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a090865-de87-442c-806a-76e91d5d374c-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:46:13.942612 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.942574    5012 generic.go:356] "Generic (PLEG): container finished" podID="8a090865-de87-442c-806a-76e91d5d374c" containerID="dd014bbb64d8d2bff6703fbf3b4e5022dd27c5703717c686e1c67310e75023ae" exitCode=0
Nov 03 00:46:13.942784 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.942632    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"
Nov 03 00:46:13.942784 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.942658    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9" event={"ID":"8a090865-de87-442c-806a-76e91d5d374c","Type":"ContainerDied","Data":"dd014bbb64d8d2bff6703fbf3b4e5022dd27c5703717c686e1c67310e75023ae"}
Nov 03 00:46:13.942784 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.942716    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9" event={"ID":"8a090865-de87-442c-806a-76e91d5d374c","Type":"ContainerDied","Data":"4a92e65840bf5c815e336c6be7ba4ca576fd047721c899dadf15f6c5c04895c6"}
Nov 03 00:46:13.942784 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.942741    5012 scope.go:122] "RemoveContainer" containerID="dd014bbb64d8d2bff6703fbf3b4e5022dd27c5703717c686e1c67310e75023ae"
Nov 03 00:46:13.968446 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.968416    5012 scope.go:122] "RemoveContainer" containerID="c910e4fab82b248bac75a840eca104a7ab622e2d9564d41f36ffbc91848e546a"
Nov 03 00:46:13.978651 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.978631    5012 scope.go:122] "RemoveContainer" containerID="dd014bbb64d8d2bff6703fbf3b4e5022dd27c5703717c686e1c67310e75023ae"
Nov 03 00:46:13.978956 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:46:13.978934    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd014bbb64d8d2bff6703fbf3b4e5022dd27c5703717c686e1c67310e75023ae\": container with ID starting with dd014bbb64d8d2bff6703fbf3b4e5022dd27c5703717c686e1c67310e75023ae not found: ID does not exist" containerID="dd014bbb64d8d2bff6703fbf3b4e5022dd27c5703717c686e1c67310e75023ae"
Nov 03 00:46:13.979018 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.978964    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd014bbb64d8d2bff6703fbf3b4e5022dd27c5703717c686e1c67310e75023ae"} err="failed to get container status \"dd014bbb64d8d2bff6703fbf3b4e5022dd27c5703717c686e1c67310e75023ae\": rpc error: code = NotFound desc = could not find container \"dd014bbb64d8d2bff6703fbf3b4e5022dd27c5703717c686e1c67310e75023ae\": container with ID starting with dd014bbb64d8d2bff6703fbf3b4e5022dd27c5703717c686e1c67310e75023ae not found: ID does not exist"
Nov 03 00:46:13.979018 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.978983    5012 scope.go:122] "RemoveContainer" containerID="c910e4fab82b248bac75a840eca104a7ab622e2d9564d41f36ffbc91848e546a"
Nov 03 00:46:13.979253 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:46:13.979225    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c910e4fab82b248bac75a840eca104a7ab622e2d9564d41f36ffbc91848e546a\": container with ID starting with c910e4fab82b248bac75a840eca104a7ab622e2d9564d41f36ffbc91848e546a not found: ID does not exist" containerID="c910e4fab82b248bac75a840eca104a7ab622e2d9564d41f36ffbc91848e546a"
Nov 03 00:46:13.979303 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:13.979252    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c910e4fab82b248bac75a840eca104a7ab622e2d9564d41f36ffbc91848e546a"} err="failed to get container status \"c910e4fab82b248bac75a840eca104a7ab622e2d9564d41f36ffbc91848e546a\": rpc error: code = NotFound desc = could not find container \"c910e4fab82b248bac75a840eca104a7ab622e2d9564d41f36ffbc91848e546a\": container with ID starting with c910e4fab82b248bac75a840eca104a7ab622e2d9564d41f36ffbc91848e546a not found: ID does not exist"
Nov 03 00:46:14.001434 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:14.001409    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"]
Nov 03 00:46:14.007488 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:14.007463    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-4fzk9"]
Nov 03 00:46:15.245043 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:46:15.245009    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a090865-de87-442c-806a-76e91d5d374c" path="/var/lib/kubelet/pods/8a090865-de87-442c-806a-76e91d5d374c/volumes"
Nov 03 00:46:37.366995 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:46:37.366897    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:47:07.365610 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:47:07.365569    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:47:37.367126 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:47:37.367080    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:48:07.369338 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:48:07.369249    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:48:21.864530 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.864495    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"]
Nov 03 00:48:21.865446 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.865422    5012 cpu_manager.go:414] "containerMap: removing container" podUID="e8cfbea4-b52d-4a09-b58d-f7f74163caf8" containerName="cleanup"
Nov 03 00:48:21.865446 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.865446    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8cfbea4-b52d-4a09-b58d-f7f74163caf8" containerName="cleanup"
Nov 03 00:48:21.865628 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.865462    5012 cpu_manager.go:414] "containerMap: removing container" podUID="6e14c921-01de-45e4-8d0c-65579eea6e32" containerName="cleanup"
Nov 03 00:48:21.865628 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.865469    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e14c921-01de-45e4-8d0c-65579eea6e32" containerName="cleanup"
Nov 03 00:48:21.865628 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.865479    5012 cpu_manager.go:414] "containerMap: removing container" podUID="f9128726-ac5c-4073-9525-ed7beca6a72f" containerName="cleanup"
Nov 03 00:48:21.865628 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.865486    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9128726-ac5c-4073-9525-ed7beca6a72f" containerName="cleanup"
Nov 03 00:48:21.865628 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.865524    5012 cpu_manager.go:414] "containerMap: removing container" podUID="8a090865-de87-442c-806a-76e91d5d374c" containerName="istio-proxy"
Nov 03 00:48:21.865628 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.865531    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a090865-de87-442c-806a-76e91d5d374c" containerName="istio-proxy"
Nov 03 00:48:21.865628 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.865540    5012 cpu_manager.go:414] "containerMap: removing container" podUID="8a5c578e-8a75-47c4-9b00-1e5e4c030488" containerName="cleanup"
Nov 03 00:48:21.865628 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.865547    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5c578e-8a75-47c4-9b00-1e5e4c030488" containerName="cleanup"
Nov 03 00:48:21.866065 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.865771    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a090865-de87-442c-806a-76e91d5d374c" containerName="istio-proxy"
Nov 03 00:48:21.866065 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.865788    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9128726-ac5c-4073-9525-ed7beca6a72f" containerName="cleanup"
Nov 03 00:48:21.866065 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.865801    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a090865-de87-442c-806a-76e91d5d374c" containerName="istio-proxy"
Nov 03 00:48:21.866065 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.865812    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e14c921-01de-45e4-8d0c-65579eea6e32" containerName="cleanup"
Nov 03 00:48:21.866065 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.865831    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5c578e-8a75-47c4-9b00-1e5e4c030488" containerName="cleanup"
Nov 03 00:48:21.869686 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.869663    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:21.883632 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.883595    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-d4d8c2-openshift-default-dockercfg-nskw8\""
Nov 03 00:48:21.899126 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.899101    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"]
Nov 03 00:48:21.967155 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.967123    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istiod-ca-cert\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:21.967290 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.967172    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-credential-socket\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:21.967335 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.967305    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxlm4\" (UniqueName: \"kubernetes.io/projected/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-kube-api-access-vxlm4\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:21.967406 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.967393    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-envoy\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:21.967467 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.967449    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-workload-certs\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:21.967515 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.967503    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-podinfo\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:21.967552 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.967534    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-token\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:21.967668 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.967650    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-data\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:21.967743 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:21.967729    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-workload-socket\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:22.068744 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:22.068718    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-workload-socket\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:22.068881 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:22.068756    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istiod-ca-cert\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:22.068881 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:22.068786    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-credential-socket\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:22.068881 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:22.068818    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxlm4\" (UniqueName: \"kubernetes.io/projected/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-kube-api-access-vxlm4\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:22.069094 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:22.068958    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-envoy\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:22.069094 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:22.069018    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-workload-certs\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:22.069094 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:22.069077    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-podinfo\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:22.069267 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:22.069110    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-token\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:22.069267 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:22.069187    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-data\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:22.069267 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:22.069208    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-workload-socket\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:22.069267 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:22.069256    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-credential-socket\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:22.069518 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:22.069386    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-workload-certs\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:22.069672 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:22.069626    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istiod-ca-cert\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:22.069825 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:22.069803    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-data\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:22.071493 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:22.071470    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-podinfo\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:22.071612 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:22.071590    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-envoy\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:22.086287 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:22.086260    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-token\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:22.086408 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:22.086325    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxlm4\" (UniqueName: \"kubernetes.io/projected/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-kube-api-access-vxlm4\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:22.184166 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:22.184113    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:22.495432 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:48:22.495399    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c4471b_fbec_4f61_8db4_3c7f2c7298c6.slice/crio-2d140c7fc5eb5d6ff1f3a0e6c32e1f0b81c0f9b1f1621b730b347b457bbfd62c WatchSource:0}: Error finding container 2d140c7fc5eb5d6ff1f3a0e6c32e1f0b81c0f9b1f1621b730b347b457bbfd62c: Status 404 returned error can't find the container with id 2d140c7fc5eb5d6ff1f3a0e6c32e1f0b81c0f9b1f1621b730b347b457bbfd62c
Nov 03 00:48:22.496100 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:22.496064    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"]
Nov 03 00:48:22.498193 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:22.498160    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:48:22.498269 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:22.498227    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:48:22.498269 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:22.498256    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:48:23.255987 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.255955    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"]
Nov 03 00:48:23.256880 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.256854    5012 cpu_manager.go:414] "containerMap: removing container" podUID="8a090865-de87-442c-806a-76e91d5d374c" containerName="istio-proxy"
Nov 03 00:48:23.256880 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.256879    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a090865-de87-442c-806a-76e91d5d374c" containerName="istio-proxy"
Nov 03 00:48:23.260785 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.260765    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.275744 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.275728    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v9b8n\""
Nov 03 00:48:23.285091 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.285064    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.285183 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.285110    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.285289 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.285266    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.285429 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.285412    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.285514 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.285486    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.285583 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.285544    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crx7m\" (UniqueName: \"kubernetes.io/projected/48052f77-fb0d-4da5-9a5b-1f24988f1bad-kube-api-access-crx7m\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.285650 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.285635    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.285754 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.285732    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.285896 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.285817    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.318485 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.318449    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"]
Nov 03 00:48:23.330280 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.330240    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg" event={"ID":"56c4471b-fbec-4f61-8db4-3c7f2c7298c6","Type":"ContainerStarted","Data":"c31e9e3590c361e26122cb575c09dd5747a9a3415b6627f99407e02290e15d57"}
Nov 03 00:48:23.330590 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.330565    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg" event={"ID":"56c4471b-fbec-4f61-8db4-3c7f2c7298c6","Type":"ContainerStarted","Data":"2d140c7fc5eb5d6ff1f3a0e6c32e1f0b81c0f9b1f1621b730b347b457bbfd62c"}
Nov 03 00:48:23.372745 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.372685    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg" podStartSLOduration=2.37266812 podStartE2EDuration="2.37266812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:23.368589921 +0000 UTC m=+2586.248173963" watchObservedRunningTime="2026-11-03 00:48:23.37266812 +0000 UTC m=+2586.252252165"
Nov 03 00:48:23.411986 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.411940    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crx7m\" (UniqueName: \"kubernetes.io/projected/48052f77-fb0d-4da5-9a5b-1f24988f1bad-kube-api-access-crx7m\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.412277 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.412249    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.412405 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.412375    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.412505 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.412476    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.412582 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.412535    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.412582 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.412564    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.412699 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.412637    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.412699 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.412657    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.413033 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.413005    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.413169 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.413144    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.413311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.413180    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.413311 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.413249    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.413899 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.413878    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.414237 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.414213    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.415765 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.415722    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.418290 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.418266    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.427804 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.427775    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"]
Nov 03 00:48:23.433606 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.433581    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crx7m\" (UniqueName: \"kubernetes.io/projected/48052f77-fb0d-4da5-9a5b-1f24988f1bad-kube-api-access-crx7m\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.434283 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.434261    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.434942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.434890    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-zhtvp\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.451033 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.451007    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"]
Nov 03 00:48:23.453324 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.453306    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-fc9316-openshift-default-dockercfg-8zk4q\""
Nov 03 00:48:23.515202 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.515106    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.515202 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.515188    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.515470 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.515247    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svglc\" (UniqueName: \"kubernetes.io/projected/8e3746c4-7a9a-470e-a4f2-4b48f8863080-kube-api-access-svglc\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.515470 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.515319    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.515470 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.515401    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.515644 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.515472    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.515644 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.515520    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.515644 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.515564    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.515818 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.515702    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.576202 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.576174    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:23.617050 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.617007    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.617215 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.617067    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.617215 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.617173    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.617215 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.617197    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.617427 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.617267    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.617427 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.617362    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.617427 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.617382    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.617427 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.617412    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svglc\" (UniqueName: \"kubernetes.io/projected/8e3746c4-7a9a-470e-a4f2-4b48f8863080-kube-api-access-svglc\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.617665 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.617456    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.617665 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.617593    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.617942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.617832    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.617942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.617879    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.618078 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.617996    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.618152 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.618119    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.619622 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.619598    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.620497 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.620477    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.635478 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.635445    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.635703 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.635684    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svglc\" (UniqueName: \"kubernetes.io/projected/8e3746c4-7a9a-470e-a4f2-4b48f8863080-kube-api-access-svglc\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.812576 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.812493    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:23.934637 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.934598    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"]
Nov 03 00:48:23.936359 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:48:23.936328    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48052f77_fb0d_4da5_9a5b_1f24988f1bad.slice/crio-3045f9eef5aaabd3917f02435a371725b71411989c54f556cf51ce1e977063f4 WatchSource:0}: Error finding container 3045f9eef5aaabd3917f02435a371725b71411989c54f556cf51ce1e977063f4: Status 404 returned error can't find the container with id 3045f9eef5aaabd3917f02435a371725b71411989c54f556cf51ce1e977063f4
Nov 03 00:48:23.939872 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.939831    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:48:23.939996 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.939935    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:48:23.939996 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:23.939978    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:48:24.126248 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:24.126209    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"]
Nov 03 00:48:24.126403 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:48:24.126288    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e3746c4_7a9a_470e_a4f2_4b48f8863080.slice/crio-4a7e432fd85cab91591af94d581f8d6889b4cca07cbd8a0c6542f6690f786d46 WatchSource:0}: Error finding container 4a7e432fd85cab91591af94d581f8d6889b4cca07cbd8a0c6542f6690f786d46: Status 404 returned error can't find the container with id 4a7e432fd85cab91591af94d581f8d6889b4cca07cbd8a0c6542f6690f786d46
Nov 03 00:48:24.130064 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:24.130027    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:48:24.130168 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:24.130106    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:48:24.130168 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:24.130147    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:48:24.184498 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:24.184473    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:24.191940 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:24.186000    5012 patch_prober.go:29] interesting pod/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body=
Nov 03 00:48:24.191940 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:24.186049    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg" podUID="56c4471b-fbec-4f61-8db4-3c7f2c7298c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused"
Nov 03 00:48:24.345426 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:24.345385    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r" event={"ID":"8e3746c4-7a9a-470e-a4f2-4b48f8863080","Type":"ContainerStarted","Data":"8435a451d41acfbc207c83134b15f7152fc83aff82e844a30ae048a2abb1d35b"}
Nov 03 00:48:24.345808 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:24.345453    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r" event={"ID":"8e3746c4-7a9a-470e-a4f2-4b48f8863080","Type":"ContainerStarted","Data":"4a7e432fd85cab91591af94d581f8d6889b4cca07cbd8a0c6542f6690f786d46"}
Nov 03 00:48:24.348924 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:24.348350    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp" event={"ID":"48052f77-fb0d-4da5-9a5b-1f24988f1bad","Type":"ContainerStarted","Data":"3859ff9d9319b35280135850e6f13bc41b75717bcd58343bf7aefc6883619bc8"}
Nov 03 00:48:24.348924 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:24.348402    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp" event={"ID":"48052f77-fb0d-4da5-9a5b-1f24988f1bad","Type":"ContainerStarted","Data":"3045f9eef5aaabd3917f02435a371725b71411989c54f556cf51ce1e977063f4"}
Nov 03 00:48:24.406379 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:24.406272    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r" podStartSLOduration=1.406252659 podStartE2EDuration="1.406252659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:24.382694348 +0000 UTC m=+2587.262278393" watchObservedRunningTime="2026-11-03 00:48:24.406252659 +0000 UTC m=+2587.285836703"
Nov 03 00:48:24.430279 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:24.430223    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp" podStartSLOduration=1.430205071 podStartE2EDuration="1.430205071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:24.429726022 +0000 UTC m=+2587.309310076" watchObservedRunningTime="2026-11-03 00:48:24.430205071 +0000 UTC m=+2587.309789114"
Nov 03 00:48:24.576298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:24.576267    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:24.813088 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:24.812983    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:25.185206 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:25.185173    5012 patch_prober.go:29] interesting pod/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body=
Nov 03 00:48:25.185476 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:25.185231    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg" podUID="56c4471b-fbec-4f61-8db4-3c7f2c7298c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused"
Nov 03 00:48:25.578998 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:25.578896    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-zhtvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body=
Nov 03 00:48:25.578998 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:25.578963    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp" podUID="48052f77-fb0d-4da5-9a5b-1f24988f1bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused"
Nov 03 00:48:25.817374 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:25.817029    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Nov 03 00:48:25.817374 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:25.817095    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r" podUID="8e3746c4-7a9a-470e-a4f2-4b48f8863080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Nov 03 00:48:26.184776 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:26.184736    5012 patch_prober.go:29] interesting pod/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body=
Nov 03 00:48:26.184982 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:26.184789    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg" podUID="56c4471b-fbec-4f61-8db4-3c7f2c7298c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused"
Nov 03 00:48:26.576905 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:26.576824    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-zhtvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body=
Nov 03 00:48:26.576905 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:26.576877    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp" podUID="48052f77-fb0d-4da5-9a5b-1f24988f1bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused"
Nov 03 00:48:26.816935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:26.813291    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Nov 03 00:48:26.816935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:26.813368    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r" podUID="8e3746c4-7a9a-470e-a4f2-4b48f8863080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Nov 03 00:48:27.184421 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:27.184389    5012 patch_prober.go:29] interesting pod/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body=
Nov 03 00:48:27.184598 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:27.184442    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg" podUID="56c4471b-fbec-4f61-8db4-3c7f2c7298c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused"
Nov 03 00:48:27.587984 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:27.585243    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-zhtvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body=
Nov 03 00:48:27.587984 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:27.585300    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp" podUID="48052f77-fb0d-4da5-9a5b-1f24988f1bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused"
Nov 03 00:48:27.845989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:27.837061    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Nov 03 00:48:27.845989 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:27.837147    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r" podUID="8e3746c4-7a9a-470e-a4f2-4b48f8863080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Nov 03 00:48:28.189352 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:28.189313    5012 patch_prober.go:29] interesting pod/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body=
Nov 03 00:48:28.189662 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:28.189624    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg" podUID="56c4471b-fbec-4f61-8db4-3c7f2c7298c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused"
Nov 03 00:48:28.579992 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:28.578456    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-zhtvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body=
Nov 03 00:48:28.579992 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:28.578520    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp" podUID="48052f77-fb0d-4da5-9a5b-1f24988f1bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused"
Nov 03 00:48:28.822932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:28.822232    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Nov 03 00:48:28.822932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:28.822293    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r" podUID="8e3746c4-7a9a-470e-a4f2-4b48f8863080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Nov 03 00:48:29.073933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.073551    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"]
Nov 03 00:48:29.107931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.107740    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.165321 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.165285    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-ac338d-openshift-default-dockercfg-hs9cf\""
Nov 03 00:48:29.249936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.212204    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istiod-ca-cert\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.257934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.254099    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-envoy\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.269930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.262077    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-workload-socket\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.331403 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.261022    5012 patch_prober.go:29] interesting pod/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused" start-of-body=
Nov 03 00:48:29.331675 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.331642    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg" podUID="56c4471b-fbec-4f61-8db4-3c7f2c7298c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.3:15021/healthz/ready\": dial tcp 10.132.1.3:15021: connect: connection refused"
Nov 03 00:48:29.335933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.331370    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn2pl\" (UniqueName: \"kubernetes.io/projected/d79c28c1-9ce7-4040-90a7-6a423b1e349e-kube-api-access-wn2pl\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.427299 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.358055    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-podinfo\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.427299 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.358147    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-token\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.427299 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.358280    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-data\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.427299 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.358312    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-workload-certs\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.427299 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.358393    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-credential-socket\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.474768 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.474725    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-workload-socket\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.547551 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.547510    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn2pl\" (UniqueName: \"kubernetes.io/projected/d79c28c1-9ce7-4040-90a7-6a423b1e349e-kube-api-access-wn2pl\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.547895 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.547875    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-podinfo\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.548074 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.548056    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-token\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.548271 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.548254    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-data\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.548379 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.548365    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-workload-certs\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.559794 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.559752    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-credential-socket\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.560217 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.560181    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istiod-ca-cert\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.575935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.560372    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-envoy\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.576728 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.500526    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-workload-socket\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.600980 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.572359    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-credential-socket\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.601213 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.572949    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-data\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.601329 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.573186    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-workload-certs\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.626000 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.589823    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"]
Nov 03 00:48:29.626262 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.591587    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istiod-ca-cert\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.626394 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.604814    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-zhtvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body=
Nov 03 00:48:29.626561 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.626534    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp" podUID="48052f77-fb0d-4da5-9a5b-1f24988f1bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused"
Nov 03 00:48:29.667637 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.665799    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-podinfo\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.725395 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.724409    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-envoy\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.756898 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.756862    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-token\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.791551 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.767559    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn2pl\" (UniqueName: \"kubernetes.io/projected/d79c28c1-9ce7-4040-90a7-6a423b1e349e-kube-api-access-wn2pl\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:29.813483 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.813054    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Nov 03 00:48:29.813483 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.813113    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r" podUID="8e3746c4-7a9a-470e-a4f2-4b48f8863080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Nov 03 00:48:29.990593 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:29.989255    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:30.246813 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:30.244868    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:30.246813 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:30.246562    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:30.380968 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:30.379495    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 00:48:30.577443 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:30.577353    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-zhtvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body=
Nov 03 00:48:30.577443 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:30.577418    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp" podUID="48052f77-fb0d-4da5-9a5b-1f24988f1bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused"
Nov 03 00:48:30.648469 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:30.648438    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"]
Nov 03 00:48:30.660634 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:30.660592    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:48:30.660741 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:30.660686    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:48:30.660741 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:30.660726    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:48:30.816931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:30.815213    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Nov 03 00:48:30.816931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:30.815265    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r" podUID="8e3746c4-7a9a-470e-a4f2-4b48f8863080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Nov 03 00:48:30.847873 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:30.844700    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv" event={"ID":"d79c28c1-9ce7-4040-90a7-6a423b1e349e","Type":"ContainerStarted","Data":"59531835c357780aece34e5bb5cf7f4a9d58149a8a62506fa890e3a617540d73"}
Nov 03 00:48:31.576789 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:31.576752    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-zhtvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body=
Nov 03 00:48:31.577317 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:31.576813    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp" podUID="48052f77-fb0d-4da5-9a5b-1f24988f1bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused"
Nov 03 00:48:31.829035 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:31.826010    5012 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Nov 03 00:48:31.829035 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:31.826075    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r" podUID="8e3746c4-7a9a-470e-a4f2-4b48f8863080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Nov 03 00:48:31.857623 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:31.857579    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv" event={"ID":"d79c28c1-9ce7-4040-90a7-6a423b1e349e","Type":"ContainerStarted","Data":"cf7739acf87ac442a550d08932dc2316973b5654f4fefb29d969764289ab5eb0"}
Nov 03 00:48:31.904029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:31.900225    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv" podStartSLOduration=2.900206259 podStartE2EDuration="2.900206259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:31.898838285 +0000 UTC m=+2594.778422356" watchObservedRunningTime="2026-11-03 00:48:31.900206259 +0000 UTC m=+2594.779790303"
Nov 03 00:48:31.992935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:31.991273    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:31.997050 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:31.997016    5012 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body=
Nov 03 00:48:31.997306 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:31.997275    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv" podUID="d79c28c1-9ce7-4040-90a7-6a423b1e349e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused"
Nov 03 00:48:32.577047 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:32.576928    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-zhtvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body=
Nov 03 00:48:32.577047 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:32.577019    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp" podUID="48052f77-fb0d-4da5-9a5b-1f24988f1bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused"
Nov 03 00:48:33.010471 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:33.010418    5012 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body=
Nov 03 00:48:33.023151 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:33.023113    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv" podUID="d79c28c1-9ce7-4040-90a7-6a423b1e349e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused"
Nov 03 00:48:33.023770 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:33.023749    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:33.025053 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:33.024860    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:33.026801 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:33.026784    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 00:48:33.994320 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:33.994282    5012 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body=
Nov 03 00:48:33.994944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:33.994891    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv" podUID="d79c28c1-9ce7-4040-90a7-6a423b1e349e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused"
Nov 03 00:48:34.269718 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:34.269625    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:34.270238 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:34.270210    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:34.271011 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:34.270989    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 00:48:34.997642 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:34.997210    5012 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body=
Nov 03 00:48:34.997642 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:34.997281    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv" podUID="d79c28c1-9ce7-4040-90a7-6a423b1e349e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused"
Nov 03 00:48:36.010936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:36.005998    5012 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body=
Nov 03 00:48:36.010936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:36.006056    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv" podUID="d79c28c1-9ce7-4040-90a7-6a423b1e349e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused"
Nov 03 00:48:36.990391 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:36.990350    5012 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body=
Nov 03 00:48:36.990593 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:36.990416    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv" podUID="d79c28c1-9ce7-4040-90a7-6a423b1e349e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused"
Nov 03 00:48:37.363287 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:48:37.363251    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:48:37.995125 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:37.995095    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:37.995564 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:37.995548    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:48:37.996284 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:48:37.996259    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 00:49:07.494831 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:49:07.494656    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:49:37.363500 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:49:37.363421    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:49:38.476944 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.476872    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"]
Nov 03 00:49:38.482962 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.482938    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.516360 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.516331    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"]
Nov 03 00:49:38.592898 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.592864    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a125db3f-e711-4e19-ac5d-cbd3827be603-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.593111 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.592905    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztwsq\" (UniqueName: \"kubernetes.io/projected/a125db3f-e711-4e19-ac5d-cbd3827be603-kube-api-access-ztwsq\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.593111 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.593076    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a125db3f-e711-4e19-ac5d-cbd3827be603-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.593213 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.593127    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a125db3f-e711-4e19-ac5d-cbd3827be603-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.593213 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.593193    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a125db3f-e711-4e19-ac5d-cbd3827be603-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.593295 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.593282    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a125db3f-e711-4e19-ac5d-cbd3827be603-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.593345 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.593319    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a125db3f-e711-4e19-ac5d-cbd3827be603-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.593383 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.593360    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a125db3f-e711-4e19-ac5d-cbd3827be603-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.593442 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.593381    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a125db3f-e711-4e19-ac5d-cbd3827be603-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.694663 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.694632    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a125db3f-e711-4e19-ac5d-cbd3827be603-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.694663 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.694669    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a125db3f-e711-4e19-ac5d-cbd3827be603-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.694892 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.694687    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a125db3f-e711-4e19-ac5d-cbd3827be603-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.694892 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.694753    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a125db3f-e711-4e19-ac5d-cbd3827be603-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.694892 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.694776    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a125db3f-e711-4e19-ac5d-cbd3827be603-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.694892 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.694836    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a125db3f-e711-4e19-ac5d-cbd3827be603-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.694892 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.694869    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a125db3f-e711-4e19-ac5d-cbd3827be603-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.695243 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.695127    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a125db3f-e711-4e19-ac5d-cbd3827be603-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.695243 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.695167    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztwsq\" (UniqueName: \"kubernetes.io/projected/a125db3f-e711-4e19-ac5d-cbd3827be603-kube-api-access-ztwsq\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.695367 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.695244    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a125db3f-e711-4e19-ac5d-cbd3827be603-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.695367 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.695165    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a125db3f-e711-4e19-ac5d-cbd3827be603-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.695367 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.695343    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a125db3f-e711-4e19-ac5d-cbd3827be603-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.695514 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.695398    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a125db3f-e711-4e19-ac5d-cbd3827be603-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.695661 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.695635    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a125db3f-e711-4e19-ac5d-cbd3827be603-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.697088 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.697053    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a125db3f-e711-4e19-ac5d-cbd3827be603-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.697321 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.697301    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a125db3f-e711-4e19-ac5d-cbd3827be603-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.712545 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.712523    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a125db3f-e711-4e19-ac5d-cbd3827be603-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.712967 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.712950    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztwsq\" (UniqueName: \"kubernetes.io/projected/a125db3f-e711-4e19-ac5d-cbd3827be603-kube-api-access-ztwsq\") pod \"maas-default-gateway-openshift-default-6cc458d887-cdvd9\" (UID: \"a125db3f-e711-4e19-ac5d-cbd3827be603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:38.800119 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:38.800062    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:39.120455 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:39.120422    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"]
Nov 03 00:49:39.121013 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:49:39.120982    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda125db3f_e711_4e19_ac5d_cbd3827be603.slice/crio-e6b39a09e9ef8ff865105bb211e61ac96d8eaa2102a605d09b0a9489875031a1 WatchSource:0}: Error finding container e6b39a09e9ef8ff865105bb211e61ac96d8eaa2102a605d09b0a9489875031a1: Status 404 returned error can't find the container with id e6b39a09e9ef8ff865105bb211e61ac96d8eaa2102a605d09b0a9489875031a1
Nov 03 00:49:39.123755 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:39.123721    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:49:39.123830 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:39.123790    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:49:39.123874 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:39.123826    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:49:39.640379 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:39.640343    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9" event={"ID":"a125db3f-e711-4e19-ac5d-cbd3827be603","Type":"ContainerStarted","Data":"62fbd0bd414cd5085bf9446b36d5f8da03641993eca63b11d5c5076bcad787cb"}
Nov 03 00:49:39.640790 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:39.640408    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9" event={"ID":"a125db3f-e711-4e19-ac5d-cbd3827be603","Type":"ContainerStarted","Data":"e6b39a09e9ef8ff865105bb211e61ac96d8eaa2102a605d09b0a9489875031a1"}
Nov 03 00:49:39.682504 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:39.682446    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9" podStartSLOduration=1.682424358 podStartE2EDuration="1.682424358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:39.679455922 +0000 UTC m=+2662.559039965" watchObservedRunningTime="2026-11-03 00:49:39.682424358 +0000 UTC m=+2662.562008400"
Nov 03 00:49:39.800900 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:39.800870    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:40.811522 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:40.811479    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-cdvd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Nov 03 00:49:40.812064 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:40.811548    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9" podUID="a125db3f-e711-4e19-ac5d-cbd3827be603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Nov 03 00:49:41.810969 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:41.808370    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-cdvd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Nov 03 00:49:41.810969 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:41.808449    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9" podUID="a125db3f-e711-4e19-ac5d-cbd3827be603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Nov 03 00:49:42.805288 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:42.805253    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-cdvd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Nov 03 00:49:42.805882 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:42.805847    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9" podUID="a125db3f-e711-4e19-ac5d-cbd3827be603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Nov 03 00:49:43.801934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:43.801395    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-cdvd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Nov 03 00:49:43.801934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:43.801466    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9" podUID="a125db3f-e711-4e19-ac5d-cbd3827be603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Nov 03 00:49:44.805931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:44.804209    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-cdvd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Nov 03 00:49:44.805931 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:44.804276    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9" podUID="a125db3f-e711-4e19-ac5d-cbd3827be603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Nov 03 00:49:45.800904 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:45.800870    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-cdvd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Nov 03 00:49:45.801117 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:45.800982    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9" podUID="a125db3f-e711-4e19-ac5d-cbd3827be603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Nov 03 00:49:46.800996 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:46.800958    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-cdvd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Nov 03 00:49:46.801577 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:46.801024    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9" podUID="a125db3f-e711-4e19-ac5d-cbd3827be603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Nov 03 00:49:48.643783 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:48.643748    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:48.644269 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:48.644088    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:49:48.645622 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:49:48.645600    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9"
Nov 03 00:50:07.363807 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:50:07.363771    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:50:08.601949 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.599271    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"]
Nov 03 00:50:08.614884 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.614832    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.650552 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.650516    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"]
Nov 03 00:50:08.715743 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.715702    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99643392-d8d1-47c9-b9d4-e87e2c837f68-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.715960 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.715766    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99643392-d8d1-47c9-b9d4-e87e2c837f68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.715960 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.715884    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99643392-d8d1-47c9-b9d4-e87e2c837f68-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.716071 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.715998    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4rqw\" (UniqueName: \"kubernetes.io/projected/99643392-d8d1-47c9-b9d4-e87e2c837f68-kube-api-access-g4rqw\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.716071 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.716025    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99643392-d8d1-47c9-b9d4-e87e2c837f68-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.716156 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.716078    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99643392-d8d1-47c9-b9d4-e87e2c837f68-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.716211 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.716183    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99643392-d8d1-47c9-b9d4-e87e2c837f68-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.716265 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.716231    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99643392-d8d1-47c9-b9d4-e87e2c837f68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.716407 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.716390    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99643392-d8d1-47c9-b9d4-e87e2c837f68-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.817288 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.817259    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99643392-d8d1-47c9-b9d4-e87e2c837f68-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.817470 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.817300    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99643392-d8d1-47c9-b9d4-e87e2c837f68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.817470 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.817317    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99643392-d8d1-47c9-b9d4-e87e2c837f68-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.817470 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.817363    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4rqw\" (UniqueName: \"kubernetes.io/projected/99643392-d8d1-47c9-b9d4-e87e2c837f68-kube-api-access-g4rqw\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.817470 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.817393    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99643392-d8d1-47c9-b9d4-e87e2c837f68-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.817470 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.817442    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99643392-d8d1-47c9-b9d4-e87e2c837f68-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.817838 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.817479    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99643392-d8d1-47c9-b9d4-e87e2c837f68-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.817838 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.817506    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99643392-d8d1-47c9-b9d4-e87e2c837f68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.817838 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.817605    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99643392-d8d1-47c9-b9d4-e87e2c837f68-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.817838 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.817716    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99643392-d8d1-47c9-b9d4-e87e2c837f68-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.817838 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.817794    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99643392-d8d1-47c9-b9d4-e87e2c837f68-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.818181 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.817973    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99643392-d8d1-47c9-b9d4-e87e2c837f68-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.818317 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.818292    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99643392-d8d1-47c9-b9d4-e87e2c837f68-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.818697 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.818674    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99643392-d8d1-47c9-b9d4-e87e2c837f68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.820038 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.820010    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99643392-d8d1-47c9-b9d4-e87e2c837f68-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.820622 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.820600    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99643392-d8d1-47c9-b9d4-e87e2c837f68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.834620 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.834596    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99643392-d8d1-47c9-b9d4-e87e2c837f68-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:08.834773 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:08.834750    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4rqw\" (UniqueName: \"kubernetes.io/projected/99643392-d8d1-47c9-b9d4-e87e2c837f68-kube-api-access-g4rqw\") pod \"maas-default-gateway-openshift-default-6cc458d887-hn6jd\" (UID: \"99643392-d8d1-47c9-b9d4-e87e2c837f68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:09.015597 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:09.015203    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:09.368201 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:50:09.368169    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99643392_d8d1_47c9_b9d4_e87e2c837f68.slice/crio-1847210005ff77702e1646af374f820119c88370b64398eab9db0b9cd0453ee9 WatchSource:0}: Error finding container 1847210005ff77702e1646af374f820119c88370b64398eab9db0b9cd0453ee9: Status 404 returned error can't find the container with id 1847210005ff77702e1646af374f820119c88370b64398eab9db0b9cd0453ee9
Nov 03 00:50:09.371635 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:09.371605    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"]
Nov 03 00:50:09.373733 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:09.373659    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:50:09.373843 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:09.373744    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:50:09.373843 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:09.373789    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:50:10.069332 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:10.069288    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd" event={"ID":"99643392-d8d1-47c9-b9d4-e87e2c837f68","Type":"ContainerStarted","Data":"3265aa5aca57732e690764b7ea727af7b265f99ba48f6819ff3052e15da327d8"}
Nov 03 00:50:10.069887 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:10.069356    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd" event={"ID":"99643392-d8d1-47c9-b9d4-e87e2c837f68","Type":"ContainerStarted","Data":"1847210005ff77702e1646af374f820119c88370b64398eab9db0b9cd0453ee9"}
Nov 03 00:50:10.124109 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:10.124053    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd" podStartSLOduration=2.124039448 podStartE2EDuration="2.124039448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:10.118224116 +0000 UTC m=+2692.997808198" watchObservedRunningTime="2026-11-03 00:50:10.124039448 +0000 UTC m=+2693.003623491"
Nov 03 00:50:11.016602 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:11.016566    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:11.018285 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:11.018256    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hn6jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Nov 03 00:50:11.018424 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:11.018310    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd" podUID="99643392-d8d1-47c9-b9d4-e87e2c837f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Nov 03 00:50:12.017004 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:12.016968    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hn6jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Nov 03 00:50:12.017497 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:12.017027    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd" podUID="99643392-d8d1-47c9-b9d4-e87e2c837f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Nov 03 00:50:13.016431 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:13.016389    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hn6jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Nov 03 00:50:13.016614 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:13.016453    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd" podUID="99643392-d8d1-47c9-b9d4-e87e2c837f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Nov 03 00:50:14.016442 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:14.016405    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hn6jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Nov 03 00:50:14.016865 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:14.016471    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd" podUID="99643392-d8d1-47c9-b9d4-e87e2c837f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Nov 03 00:50:15.016426 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:15.016390    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hn6jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Nov 03 00:50:15.016612 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:15.016459    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd" podUID="99643392-d8d1-47c9-b9d4-e87e2c837f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Nov 03 00:50:16.017173 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:16.017143    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hn6jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Nov 03 00:50:16.017564 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:16.017198    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd" podUID="99643392-d8d1-47c9-b9d4-e87e2c837f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Nov 03 00:50:17.016313 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:17.016273    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hn6jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body=
Nov 03 00:50:17.016532 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:17.016334    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd" podUID="99643392-d8d1-47c9-b9d4-e87e2c837f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused"
Nov 03 00:50:18.296720 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:18.296685    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 00:50:18.298020 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:18.297995    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 00:50:18.303294 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:18.303273    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:50:18.304453 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:18.304427    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:50:18.791865 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:18.791831    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:18.792336 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:18.792271    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:18.793239 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:50:18.793214    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd"
Nov 03 00:50:37.365050 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:50:37.365008    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:51:07.421251 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:51:07.421147    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:51:37.369277 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:51:37.369233    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:52:07.368505 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:52:07.368455    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:52:37.365763 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:52:37.365683    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:53:07.363776 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:53:07.363739    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:53:37.363983 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:53:37.363949    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:53:58.550450 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.550361    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"]
Nov 03 00:53:58.556558 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.556531    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.577049 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.577026    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"]
Nov 03 00:53:58.582238 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.582214    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-726205-openshift-default-dockercfg-nzntw\""
Nov 03 00:53:58.685497 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.685459    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-workload-certs\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.685654 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.685505    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istiod-ca-cert\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.685654 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.685561    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-podinfo\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.685654 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.685589    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6885\" (UniqueName: \"kubernetes.io/projected/8e95dc58-cf96-48fd-91b0-7ce6156ed573-kube-api-access-s6885\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.685770 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.685679    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-workload-socket\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.685770 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.685733    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-credential-socket\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.685838 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.685813    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-envoy\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.685838 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.685832    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-token\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.685941 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.685870    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-data\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.796943 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.792009    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-workload-certs\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.809750 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.809675    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istiod-ca-cert\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.815055 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.815028    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-podinfo\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.815750 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.815728    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6885\" (UniqueName: \"kubernetes.io/projected/8e95dc58-cf96-48fd-91b0-7ce6156ed573-kube-api-access-s6885\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.825374 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.825348    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-workload-socket\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.825945 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.825894    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-credential-socket\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.840932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.826433    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-envoy\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.841659 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.841636    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-token\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.841883 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.841860    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-data\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.842273 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.826336    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-credential-socket\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.859227 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.797586    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-workload-certs\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.891853 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.810865    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istiod-ca-cert\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.892302 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.825801    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-workload-socket\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.892462 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.842685    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-data\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.892574 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.860654    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-envoy\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.892677 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.862436    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-podinfo\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.894043 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.893997    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6885\" (UniqueName: \"kubernetes.io/projected/8e95dc58-cf96-48fd-91b0-7ce6156ed573-kube-api-access-s6885\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:58.897853 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:58.897810    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-token\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:59.189439 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:59.189396    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:53:59.525400 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:53:59.525373    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e95dc58_cf96_48fd_91b0_7ce6156ed573.slice/crio-0d6c118769bf95a0b1bd50800f2f11fa74df48c2d0df7d3c850d0c569c62207b WatchSource:0}: Error finding container 0d6c118769bf95a0b1bd50800f2f11fa74df48c2d0df7d3c850d0c569c62207b: Status 404 returned error can't find the container with id 0d6c118769bf95a0b1bd50800f2f11fa74df48c2d0df7d3c850d0c569c62207b
Nov 03 00:53:59.525646 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:59.525626    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"]
Nov 03 00:53:59.528141 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:59.528109    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:53:59.528235 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:59.528171    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:53:59.528235 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:53:59.528200    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:54:00.306113 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:00.306078    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2" event={"ID":"8e95dc58-cf96-48fd-91b0-7ce6156ed573","Type":"ContainerStarted","Data":"e3e6fa7439f7bcdc58bae33503ffb76ffc5e00cf45aefd32267ee44e7c1cbb7d"}
Nov 03 00:54:00.306638 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:00.306139    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2" event={"ID":"8e95dc58-cf96-48fd-91b0-7ce6156ed573","Type":"ContainerStarted","Data":"0d6c118769bf95a0b1bd50800f2f11fa74df48c2d0df7d3c850d0c569c62207b"}
Nov 03 00:54:01.190766 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:01.190724    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:54:01.197875 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:01.197845    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:54:01.248359 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:01.248300    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2" podStartSLOduration=3.248285842 podStartE2EDuration="3.248285842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:00.364165754 +0000 UTC m=+2923.243749796" watchObservedRunningTime="2026-11-03 00:54:01.248285842 +0000 UTC m=+2924.127869885"
Nov 03 00:54:01.314064 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:01.314027    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:54:01.314940 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:01.314890    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:54:07.367545 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:54:07.367508    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:54:37.541536 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:54:37.541488    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:54:54.042274 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:54.042239    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9"]
Nov 03 00:54:54.047336 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:54.047315    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9"
Nov 03 00:54:54.065663 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:54.065642    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9"]
Nov 03 00:54:54.070066 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:54.070023    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-883a9e\""
Nov 03 00:54:54.070464 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:54.070443    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-883a9e-dockercfg-j72pp\""
Nov 03 00:54:54.109649 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:54.109616    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq"]
Nov 03 00:54:54.114282 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:54.114262    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq"
Nov 03 00:54:54.126698 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:54.126676    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8216aa2d-f296-4d63-ad6b-62e2aa583c5c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9\" (UID: \"8216aa2d-f296-4d63-ad6b-62e2aa583c5c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9"
Nov 03 00:54:54.126972 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:54.126950    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twqdd\" (UniqueName: \"kubernetes.io/projected/b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3-kube-api-access-twqdd\") pod \"payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq\" (UID: \"b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq"
Nov 03 00:54:54.127191 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:54.127171    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjfjb\" (UniqueName: \"kubernetes.io/projected/8216aa2d-f296-4d63-ad6b-62e2aa583c5c-kube-api-access-kjfjb\") pod \"payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9\" (UID: \"8216aa2d-f296-4d63-ad6b-62e2aa583c5c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9"
Nov 03 00:54:54.127320 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:54.127255    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq\" (UID: \"b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq"
Nov 03 00:54:54.133165 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:54.133144    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq"]
Nov 03 00:54:54.266930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:54.259926    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8216aa2d-f296-4d63-ad6b-62e2aa583c5c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9\" (UID: \"8216aa2d-f296-4d63-ad6b-62e2aa583c5c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9"
Nov 03 00:54:54.266930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:54.259988    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twqdd\" (UniqueName: \"kubernetes.io/projected/b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3-kube-api-access-twqdd\") pod \"payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq\" (UID: \"b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq"
Nov 03 00:54:54.266930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:54.260087    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjfjb\" (UniqueName: \"kubernetes.io/projected/8216aa2d-f296-4d63-ad6b-62e2aa583c5c-kube-api-access-kjfjb\") pod \"payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9\" (UID: \"8216aa2d-f296-4d63-ad6b-62e2aa583c5c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9"
Nov 03 00:54:54.266930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:54.260171    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq\" (UID: \"b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq"
Nov 03 00:54:54.266930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:54.261330    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq\" (UID: \"b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq"
Nov 03 00:54:54.266930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:54.262139    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8216aa2d-f296-4d63-ad6b-62e2aa583c5c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9\" (UID: \"8216aa2d-f296-4d63-ad6b-62e2aa583c5c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9"
Nov 03 00:54:54.428303 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:54.428267    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twqdd\" (UniqueName: \"kubernetes.io/projected/b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3-kube-api-access-twqdd\") pod \"payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq\" (UID: \"b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq"
Nov 03 00:54:54.436603 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:54.433511    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq"
Nov 03 00:54:54.443428 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:54.438693    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjfjb\" (UniqueName: \"kubernetes.io/projected/8216aa2d-f296-4d63-ad6b-62e2aa583c5c-kube-api-access-kjfjb\") pod \"payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9\" (UID: \"8216aa2d-f296-4d63-ad6b-62e2aa583c5c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9"
Nov 03 00:54:54.662879 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:54.662848    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9"
Nov 03 00:54:54.766185 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:54.766145    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq"]
Nov 03 00:54:54.771044 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:54:54.771015    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8cf0370_bd15_4ea1_9aa2_e83ce4938ea3.slice/crio-6ff8e01204bdd345e19ca6ce04d6f2f50e188975e70a11ebe84c6b22207c4095 WatchSource:0}: Error finding container 6ff8e01204bdd345e19ca6ce04d6f2f50e188975e70a11ebe84c6b22207c4095: Status 404 returned error can't find the container with id 6ff8e01204bdd345e19ca6ce04d6f2f50e188975e70a11ebe84c6b22207c4095
Nov 03 00:54:55.008734 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:54:55.008704    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8216aa2d_f296_4d63_ad6b_62e2aa583c5c.slice/crio-6000608dd5131cc76ca234bd21104d9a5dbf5267f23b69d1b9b512659c4ccaad WatchSource:0}: Error finding container 6000608dd5131cc76ca234bd21104d9a5dbf5267f23b69d1b9b512659c4ccaad: Status 404 returned error can't find the container with id 6000608dd5131cc76ca234bd21104d9a5dbf5267f23b69d1b9b512659c4ccaad
Nov 03 00:54:55.008945 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:55.008919    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9"]
Nov 03 00:54:55.042218 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:55.042190    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9" event={"ID":"8216aa2d-f296-4d63-ad6b-62e2aa583c5c","Type":"ContainerStarted","Data":"6000608dd5131cc76ca234bd21104d9a5dbf5267f23b69d1b9b512659c4ccaad"}
Nov 03 00:54:55.043928 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:55.043878    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq" event={"ID":"b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3","Type":"ContainerStarted","Data":"21bdf632fdeb4c117c30a4003951d9b4c8634ba06005d4694f3d7643c2685f94"}
Nov 03 00:54:55.044275 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:55.044019    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq" event={"ID":"b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3","Type":"ContainerStarted","Data":"6ff8e01204bdd345e19ca6ce04d6f2f50e188975e70a11ebe84c6b22207c4095"}
Nov 03 00:54:55.044275 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:55.044139    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq"
Nov 03 00:54:55.080728 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:55.080675    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq" podStartSLOduration=1.08065862 podStartE2EDuration="1.08065862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:55.078827411 +0000 UTC m=+2977.958411454" watchObservedRunningTime="2026-11-03 00:54:55.08065862 +0000 UTC m=+2977.960242664"
Nov 03 00:54:56.054758 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:56.054717    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9" event={"ID":"8216aa2d-f296-4d63-ad6b-62e2aa583c5c","Type":"ContainerStarted","Data":"2c43ee91523985f544219addd6bb267746c4c47490909c8c7d43af93b8d35506"}
Nov 03 00:54:56.055262 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:56.054783    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9"
Nov 03 00:54:56.093068 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:54:56.093016    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9" podStartSLOduration=2.093003145 podStartE2EDuration="2.093003145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:56.089395928 +0000 UTC m=+2978.968979983" watchObservedRunningTime="2026-11-03 00:54:56.093003145 +0000 UTC m=+2978.972587193"
Nov 03 00:55:00.600936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:00.600385    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575"]
Nov 03 00:55:00.666684 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:00.663117    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575"]
Nov 03 00:55:00.666684 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:00.663246    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575"
Nov 03 00:55:00.683190 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:00.683010    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c738e0\"/\"openshift-service-ca.crt\""
Nov 03 00:55:00.693327 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:00.693289    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c738e0\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c738e0\""
Nov 03 00:55:00.693611 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:00.693594    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c738e0\"/\"kube-root-ca.crt\""
Nov 03 00:55:00.693859 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:00.693844    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c738e0\"/\"routing-overlay\""
Nov 03 00:55:00.694128 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:00.694111    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c738e0\"/\"payload-processing-external-model-e2e--bf641009-dockercfg-hdkvs\""
Nov 03 00:55:00.694386 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:00.694354    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c738e0\"/\"e2e-cfl-b3e08e-provider-cred\""
Nov 03 00:55:00.796866 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:00.796826    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d75cedc9-dc00-4879-8a9b-c7625d45a94d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575\" (UID: \"d75cedc9-dc00-4879-8a9b-c7625d45a94d\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575"
Nov 03 00:55:00.834587 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:00.822962    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d75cedc9-dc00-4879-8a9b-c7625d45a94d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575\" (UID: \"d75cedc9-dc00-4879-8a9b-c7625d45a94d\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575"
Nov 03 00:55:00.834587 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:00.830141    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d75cedc9-dc00-4879-8a9b-c7625d45a94d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575\" (UID: \"d75cedc9-dc00-4879-8a9b-c7625d45a94d\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575"
Nov 03 00:55:00.834587 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:00.830208    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgxmm\" (UniqueName: \"kubernetes.io/projected/d75cedc9-dc00-4879-8a9b-c7625d45a94d-kube-api-access-cgxmm\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575\" (UID: \"d75cedc9-dc00-4879-8a9b-c7625d45a94d\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575"
Nov 03 00:55:00.935774 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:00.935678    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d75cedc9-dc00-4879-8a9b-c7625d45a94d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575\" (UID: \"d75cedc9-dc00-4879-8a9b-c7625d45a94d\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575"
Nov 03 00:55:00.935978 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:00.935834    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d75cedc9-dc00-4879-8a9b-c7625d45a94d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575\" (UID: \"d75cedc9-dc00-4879-8a9b-c7625d45a94d\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575"
Nov 03 00:55:00.935978 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:00.935947    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d75cedc9-dc00-4879-8a9b-c7625d45a94d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575\" (UID: \"d75cedc9-dc00-4879-8a9b-c7625d45a94d\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575"
Nov 03 00:55:00.936130 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:00.936011    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgxmm\" (UniqueName: \"kubernetes.io/projected/d75cedc9-dc00-4879-8a9b-c7625d45a94d-kube-api-access-cgxmm\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575\" (UID: \"d75cedc9-dc00-4879-8a9b-c7625d45a94d\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575"
Nov 03 00:55:00.952007 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:00.951695    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d75cedc9-dc00-4879-8a9b-c7625d45a94d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575\" (UID: \"d75cedc9-dc00-4879-8a9b-c7625d45a94d\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575"
Nov 03 00:55:00.959034 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:00.952795    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d75cedc9-dc00-4879-8a9b-c7625d45a94d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575\" (UID: \"d75cedc9-dc00-4879-8a9b-c7625d45a94d\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575"
Nov 03 00:55:00.959511 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:00.959484    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgxmm\" (UniqueName: \"kubernetes.io/projected/d75cedc9-dc00-4879-8a9b-c7625d45a94d-kube-api-access-cgxmm\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575\" (UID: \"d75cedc9-dc00-4879-8a9b-c7625d45a94d\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575"
Nov 03 00:55:00.960263 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:00.960236    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d75cedc9-dc00-4879-8a9b-c7625d45a94d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575\" (UID: \"d75cedc9-dc00-4879-8a9b-c7625d45a94d\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575"
Nov 03 00:55:01.000079 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:01.000044    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575"
Nov 03 00:55:01.312762 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:01.312732    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575"]
Nov 03 00:55:01.314058 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:55:01.314028    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75cedc9_dc00_4879_8a9b_c7625d45a94d.slice/crio-429486f9870fb5eed3c167a97a58772812e038bdba31ce0ee940e7c5d5a6173c WatchSource:0}: Error finding container 429486f9870fb5eed3c167a97a58772812e038bdba31ce0ee940e7c5d5a6173c: Status 404 returned error can't find the container with id 429486f9870fb5eed3c167a97a58772812e038bdba31ce0ee940e7c5d5a6173c
Nov 03 00:55:02.111969 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:02.111935    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575" event={"ID":"d75cedc9-dc00-4879-8a9b-c7625d45a94d","Type":"ContainerStarted","Data":"374e1627c45a4e9e78409128f7fd312433ed813091aad646cc2f2e371fb0410c"}
Nov 03 00:55:02.112402 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:02.111991    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575" event={"ID":"d75cedc9-dc00-4879-8a9b-c7625d45a94d","Type":"ContainerStarted","Data":"429486f9870fb5eed3c167a97a58772812e038bdba31ce0ee940e7c5d5a6173c"}
Nov 03 00:55:02.112402 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:02.112109    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575"
Nov 03 00:55:02.150116 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:02.150074    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575" podStartSLOduration=2.150061475 podStartE2EDuration="2.150061475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:02.146386294 +0000 UTC m=+2985.025970340" watchObservedRunningTime="2026-11-03 00:55:02.150061475 +0000 UTC m=+2985.029645517"
Nov 03 00:55:06.059061 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:06.059021    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq"
Nov 03 00:55:07.066459 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:07.066426    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9"
Nov 03 00:55:07.363631 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:55:07.363593    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:55:09.740298 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:09.740265    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-0f730e/e2e-usage-sim-deff0d-7b94db74b-ndvxm"]
Nov 03 00:55:09.745280 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:09.745255    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-0f730e/e2e-usage-sim-deff0d-7b94db74b-ndvxm"
Nov 03 00:55:09.760200 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:09.760177    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-0f730e/e2e-usage-sim-deff0d-7b94db74b-ndvxm"]
Nov 03 00:55:09.765561 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:09.765540    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-0f730e\"/\"kube-root-ca.crt\""
Nov 03 00:55:09.768685 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:09.768635    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-0f730e\"/\"openshift-service-ca.crt\""
Nov 03 00:55:09.768883 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:09.768859    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-0f730e\"/\"e2e-usage-sim-deff0d-cert\""
Nov 03 00:55:09.768883 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:09.768869    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-0f730e\"/\"default-dockercfg-k9vc6\""
Nov 03 00:55:09.826021 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:09.825990    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1b79e717-6fed-4b89-9d64-783211f4f037-tls\") pod \"e2e-usage-sim-deff0d-7b94db74b-ndvxm\" (UID: \"1b79e717-6fed-4b89-9d64-783211f4f037\") " pod="e2e-extmod-shr-user-0f730e/e2e-usage-sim-deff0d-7b94db74b-ndvxm"
Nov 03 00:55:09.826213 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:09.826195    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thwpt\" (UniqueName: \"kubernetes.io/projected/1b79e717-6fed-4b89-9d64-783211f4f037-kube-api-access-thwpt\") pod \"e2e-usage-sim-deff0d-7b94db74b-ndvxm\" (UID: \"1b79e717-6fed-4b89-9d64-783211f4f037\") " pod="e2e-extmod-shr-user-0f730e/e2e-usage-sim-deff0d-7b94db74b-ndvxm"
Nov 03 00:55:09.927333 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:09.927300    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1b79e717-6fed-4b89-9d64-783211f4f037-tls\") pod \"e2e-usage-sim-deff0d-7b94db74b-ndvxm\" (UID: \"1b79e717-6fed-4b89-9d64-783211f4f037\") " pod="e2e-extmod-shr-user-0f730e/e2e-usage-sim-deff0d-7b94db74b-ndvxm"
Nov 03 00:55:09.927508 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:09.927433    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thwpt\" (UniqueName: \"kubernetes.io/projected/1b79e717-6fed-4b89-9d64-783211f4f037-kube-api-access-thwpt\") pod \"e2e-usage-sim-deff0d-7b94db74b-ndvxm\" (UID: \"1b79e717-6fed-4b89-9d64-783211f4f037\") " pod="e2e-extmod-shr-user-0f730e/e2e-usage-sim-deff0d-7b94db74b-ndvxm"
Nov 03 00:55:09.929887 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:09.929862    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1b79e717-6fed-4b89-9d64-783211f4f037-tls\") pod \"e2e-usage-sim-deff0d-7b94db74b-ndvxm\" (UID: \"1b79e717-6fed-4b89-9d64-783211f4f037\") " pod="e2e-extmod-shr-user-0f730e/e2e-usage-sim-deff0d-7b94db74b-ndvxm"
Nov 03 00:55:09.944471 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:09.944443    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thwpt\" (UniqueName: \"kubernetes.io/projected/1b79e717-6fed-4b89-9d64-783211f4f037-kube-api-access-thwpt\") pod \"e2e-usage-sim-deff0d-7b94db74b-ndvxm\" (UID: \"1b79e717-6fed-4b89-9d64-783211f4f037\") " pod="e2e-extmod-shr-user-0f730e/e2e-usage-sim-deff0d-7b94db74b-ndvxm"
Nov 03 00:55:10.060231 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:10.060173    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-0f730e/e2e-usage-sim-deff0d-7b94db74b-ndvxm"
Nov 03 00:55:10.361068 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:55:10.361040    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b79e717_6fed_4b89_9d64_783211f4f037.slice/crio-4ce44c88d89d53c1172b0597a6815eb488a301798e5e57b4a04bc7115437f0f8 WatchSource:0}: Error finding container 4ce44c88d89d53c1172b0597a6815eb488a301798e5e57b4a04bc7115437f0f8: Status 404 returned error can't find the container with id 4ce44c88d89d53c1172b0597a6815eb488a301798e5e57b4a04bc7115437f0f8
Nov 03 00:55:10.361695 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:10.361669    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-0f730e/e2e-usage-sim-deff0d-7b94db74b-ndvxm"]
Nov 03 00:55:11.275179 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:11.274892    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-0f730e/e2e-usage-sim-deff0d-7b94db74b-ndvxm" event={"ID":"1b79e717-6fed-4b89-9d64-783211f4f037","Type":"ContainerStarted","Data":"2901a8c50fec96260010476a7713bf40bc913c97649401eb96689fdcfe1427aa"}
Nov 03 00:55:11.275179 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:11.274974    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-0f730e/e2e-usage-sim-deff0d-7b94db74b-ndvxm"
Nov 03 00:55:11.275179 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:11.274989    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-0f730e/e2e-usage-sim-deff0d-7b94db74b-ndvxm" event={"ID":"1b79e717-6fed-4b89-9d64-783211f4f037","Type":"ContainerStarted","Data":"4ce44c88d89d53c1172b0597a6815eb488a301798e5e57b4a04bc7115437f0f8"}
Nov 03 00:55:11.280017 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:11.279982    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-0f730e/e2e-usage-sim-deff0d-7b94db74b-ndvxm"
Nov 03 00:55:11.315037 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:11.314993    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-0f730e/e2e-usage-sim-deff0d-7b94db74b-ndvxm" podStartSLOduration=2.314980295 podStartE2EDuration="2.314980295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:11.309809451 +0000 UTC m=+2994.189393495" watchObservedRunningTime="2026-11-03 00:55:11.314980295 +0000 UTC m=+2994.194564360"
Nov 03 00:55:12.145391 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:12.145361    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575"]
Nov 03 00:55:12.148547 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:12.148522    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575"
Nov 03 00:55:17.145680 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:17.145616    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575" podUID="d75cedc9-dc00-4879-8a9b-c7625d45a94d" containerName="payload-processing" containerID="cri-o://374e1627c45a4e9e78409128f7fd312433ed813091aad646cc2f2e371fb0410c" gracePeriod=25
Nov 03 00:55:17.493049 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:17.493017    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575"
Nov 03 00:55:17.495493 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:17.495462    5012 generic.go:356] "Generic (PLEG): container finished" podID="d75cedc9-dc00-4879-8a9b-c7625d45a94d" containerID="374e1627c45a4e9e78409128f7fd312433ed813091aad646cc2f2e371fb0410c" exitCode=0
Nov 03 00:55:17.495659 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:17.495505    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575"
Nov 03 00:55:17.495659 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:17.495551    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575" event={"ID":"d75cedc9-dc00-4879-8a9b-c7625d45a94d","Type":"ContainerDied","Data":"374e1627c45a4e9e78409128f7fd312433ed813091aad646cc2f2e371fb0410c"}
Nov 03 00:55:17.495659 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:17.495621    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575" event={"ID":"d75cedc9-dc00-4879-8a9b-c7625d45a94d","Type":"ContainerDied","Data":"429486f9870fb5eed3c167a97a58772812e038bdba31ce0ee940e7c5d5a6173c"}
Nov 03 00:55:17.495659 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:17.495650    5012 scope.go:122] "RemoveContainer" containerID="374e1627c45a4e9e78409128f7fd312433ed813091aad646cc2f2e371fb0410c"
Nov 03 00:55:17.507463 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:17.507442    5012 scope.go:122] "RemoveContainer" containerID="374e1627c45a4e9e78409128f7fd312433ed813091aad646cc2f2e371fb0410c"
Nov 03 00:55:17.507711 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:55:17.507693    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"374e1627c45a4e9e78409128f7fd312433ed813091aad646cc2f2e371fb0410c\": container with ID starting with 374e1627c45a4e9e78409128f7fd312433ed813091aad646cc2f2e371fb0410c not found: ID does not exist" containerID="374e1627c45a4e9e78409128f7fd312433ed813091aad646cc2f2e371fb0410c"
Nov 03 00:55:17.507764 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:17.507718    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"374e1627c45a4e9e78409128f7fd312433ed813091aad646cc2f2e371fb0410c"} err="failed to get container status \"374e1627c45a4e9e78409128f7fd312433ed813091aad646cc2f2e371fb0410c\": rpc error: code = NotFound desc = could not find container \"374e1627c45a4e9e78409128f7fd312433ed813091aad646cc2f2e371fb0410c\": container with ID starting with 374e1627c45a4e9e78409128f7fd312433ed813091aad646cc2f2e371fb0410c not found: ID does not exist"
Nov 03 00:55:17.597794 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:17.597765    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d75cedc9-dc00-4879-8a9b-c7625d45a94d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d75cedc9-dc00-4879-8a9b-c7625d45a94d-provider-credentials\") pod \"d75cedc9-dc00-4879-8a9b-c7625d45a94d\" (UID: \"d75cedc9-dc00-4879-8a9b-c7625d45a94d\") "
Nov 03 00:55:17.597993 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:17.597804    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d75cedc9-dc00-4879-8a9b-c7625d45a94d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d75cedc9-dc00-4879-8a9b-c7625d45a94d-routing-overlay-volume\") pod \"d75cedc9-dc00-4879-8a9b-c7625d45a94d\" (UID: \"d75cedc9-dc00-4879-8a9b-c7625d45a94d\") "
Nov 03 00:55:17.597993 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:17.597848    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d75cedc9-dc00-4879-8a9b-c7625d45a94d-kube-api-access-cgxmm\" (UniqueName: \"kubernetes.io/projected/d75cedc9-dc00-4879-8a9b-c7625d45a94d-kube-api-access-cgxmm\") pod \"d75cedc9-dc00-4879-8a9b-c7625d45a94d\" (UID: \"d75cedc9-dc00-4879-8a9b-c7625d45a94d\") "
Nov 03 00:55:17.597993 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:17.597936    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d75cedc9-dc00-4879-8a9b-c7625d45a94d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d75cedc9-dc00-4879-8a9b-c7625d45a94d-plugins-config-volume\") pod \"d75cedc9-dc00-4879-8a9b-c7625d45a94d\" (UID: \"d75cedc9-dc00-4879-8a9b-c7625d45a94d\") "
Nov 03 00:55:17.598326 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:17.598300    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d75cedc9-dc00-4879-8a9b-c7625d45a94d-routing-overlay-volume" pod "d75cedc9-dc00-4879-8a9b-c7625d45a94d" (UID: "d75cedc9-dc00-4879-8a9b-c7625d45a94d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:55:17.598392 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:17.598353    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d75cedc9-dc00-4879-8a9b-c7625d45a94d-plugins-config-volume" pod "d75cedc9-dc00-4879-8a9b-c7625d45a94d" (UID: "d75cedc9-dc00-4879-8a9b-c7625d45a94d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:55:17.598546 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:17.598527    5012 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d75cedc9-dc00-4879-8a9b-c7625d45a94d-routing-overlay-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:55:17.598613 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:17.598551    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d75cedc9-dc00-4879-8a9b-c7625d45a94d-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:55:17.600224 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:17.600204    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75cedc9-dc00-4879-8a9b-c7625d45a94d-provider-credentials" pod "d75cedc9-dc00-4879-8a9b-c7625d45a94d" (UID: "d75cedc9-dc00-4879-8a9b-c7625d45a94d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:55:17.600331 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:17.600230    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75cedc9-dc00-4879-8a9b-c7625d45a94d-kube-api-access-cgxmm" pod "d75cedc9-dc00-4879-8a9b-c7625d45a94d" (UID: "d75cedc9-dc00-4879-8a9b-c7625d45a94d"). InnerVolumeSpecName "kube-api-access-cgxmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:55:17.700566 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:17.700504    5012 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d75cedc9-dc00-4879-8a9b-c7625d45a94d-provider-credentials\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:55:17.700566 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:17.700538    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgxmm\" (UniqueName: \"kubernetes.io/projected/d75cedc9-dc00-4879-8a9b-c7625d45a94d-kube-api-access-cgxmm\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:55:17.843662 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:17.843631    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575"]
Nov 03 00:55:17.850667 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:17.850643    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57fc575"]
Nov 03 00:55:18.368820 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:18.368796    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 00:55:18.371947 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:18.371922    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 00:55:18.373575 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:18.373553    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:55:18.376469 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:18.376453    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 00:55:19.246782 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:19.246747    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d75cedc9-dc00-4879-8a9b-c7625d45a94d" path="/var/lib/kubelet/pods/d75cedc9-dc00-4879-8a9b-c7625d45a94d/volumes"
Nov 03 00:55:37.485595 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:55:37.485515    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:55:43.554413 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:43.554375    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v"]
Nov 03 00:55:43.555275 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:43.555256    5012 cpu_manager.go:414] "containerMap: removing container" podUID="d75cedc9-dc00-4879-8a9b-c7625d45a94d" containerName="payload-processing"
Nov 03 00:55:43.555331 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:43.555278    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d75cedc9-dc00-4879-8a9b-c7625d45a94d" containerName="payload-processing"
Nov 03 00:55:43.555411 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:43.555402    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="d75cedc9-dc00-4879-8a9b-c7625d45a94d" containerName="payload-processing"
Nov 03 00:55:43.559237 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:43.559220    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v"
Nov 03 00:55:43.578416 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:43.578391    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v"]
Nov 03 00:55:43.582891 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:43.582845    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-883a9e-serving-cert\""
Nov 03 00:55:43.646674 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:43.646632    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-maas-api-tls\") pod \"maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v\" (UID: \"4d5d2f80-69a4-45d5-90e1-2a8335e38cda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v"
Nov 03 00:55:43.646964 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:43.646937    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-metrics-certs\") pod \"maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v\" (UID: \"4d5d2f80-69a4-45d5-90e1-2a8335e38cda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v"
Nov 03 00:55:43.647253 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:43.647231    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gm5s\" (UniqueName: \"kubernetes.io/projected/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-kube-api-access-6gm5s\") pod \"maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v\" (UID: \"4d5d2f80-69a4-45d5-90e1-2a8335e38cda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v"
Nov 03 00:55:43.647441 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:43.647424    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-openshift-service-ca\") pod \"maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v\" (UID: \"4d5d2f80-69a4-45d5-90e1-2a8335e38cda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v"
Nov 03 00:55:43.749510 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:43.749470    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-maas-api-tls\") pod \"maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v\" (UID: \"4d5d2f80-69a4-45d5-90e1-2a8335e38cda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v"
Nov 03 00:55:43.749510 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:43.749513    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-metrics-certs\") pod \"maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v\" (UID: \"4d5d2f80-69a4-45d5-90e1-2a8335e38cda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v"
Nov 03 00:55:43.749774 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:43.749628    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gm5s\" (UniqueName: \"kubernetes.io/projected/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-kube-api-access-6gm5s\") pod \"maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v\" (UID: \"4d5d2f80-69a4-45d5-90e1-2a8335e38cda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v"
Nov 03 00:55:43.749774 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:43.749680    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-openshift-service-ca\") pod \"maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v\" (UID: \"4d5d2f80-69a4-45d5-90e1-2a8335e38cda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v"
Nov 03 00:55:43.750437 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:43.750408    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-openshift-service-ca\") pod \"maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v\" (UID: \"4d5d2f80-69a4-45d5-90e1-2a8335e38cda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v"
Nov 03 00:55:43.752101 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:43.752074    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-maas-api-tls\") pod \"maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v\" (UID: \"4d5d2f80-69a4-45d5-90e1-2a8335e38cda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v"
Nov 03 00:55:43.752312 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:43.752216    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-metrics-certs\") pod \"maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v\" (UID: \"4d5d2f80-69a4-45d5-90e1-2a8335e38cda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v"
Nov 03 00:55:43.767288 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:43.767264    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gm5s\" (UniqueName: \"kubernetes.io/projected/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-kube-api-access-6gm5s\") pod \"maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v\" (UID: \"4d5d2f80-69a4-45d5-90e1-2a8335e38cda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v"
Nov 03 00:55:43.878833 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:43.878798    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v"
Nov 03 00:55:44.212807 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:55:44.212775    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d5d2f80_69a4_45d5_90e1_2a8335e38cda.slice/crio-a08690fed43237383f48f5658d63195edbca21a928e153690257c7956300472d WatchSource:0}: Error finding container a08690fed43237383f48f5658d63195edbca21a928e153690257c7956300472d: Status 404 returned error can't find the container with id a08690fed43237383f48f5658d63195edbca21a928e153690257c7956300472d
Nov 03 00:55:44.214350 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:44.214331    5012 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:55:44.215508 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:44.215479    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v"]
Nov 03 00:55:44.984573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:44.984485    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v" event={"ID":"4d5d2f80-69a4-45d5-90e1-2a8335e38cda","Type":"ContainerStarted","Data":"bf08357aeede1fd002cf857c0ac385d4ee09a208d63313a362fa26e8333b5b12"}
Nov 03 00:55:44.984573 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:44.984563    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v"
Nov 03 00:55:44.984979 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:44.984590    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v" event={"ID":"4d5d2f80-69a4-45d5-90e1-2a8335e38cda","Type":"ContainerStarted","Data":"a08690fed43237383f48f5658d63195edbca21a928e153690257c7956300472d"}
Nov 03 00:55:45.027570 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:45.027516    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v" podStartSLOduration=1.636102733 podStartE2EDuration="2.027503197s" totalImagesPullingTime="391.400464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:45.021682438 +0000 UTC m=+3027.901266481" watchObservedRunningTime="2026-11-03 00:55:45.027503197 +0000 UTC m=+3027.907087260"
Nov 03 00:55:47.364300 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:47.355203    5012 patch_prober.go:29] interesting pod/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:55:47.364300 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:47.355295    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2" podUID="8e95dc58-cf96-48fd-91b0-7ce6156ed573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:49.051516 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:49.051478    5012 generic.go:356] "Generic (PLEG): container finished" podID="8e95dc58-cf96-48fd-91b0-7ce6156ed573" containerID="e3e6fa7439f7bcdc58bae33503ffb76ffc5e00cf45aefd32267ee44e7c1cbb7d" exitCode=0
Nov 03 00:55:49.052072 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:49.051554    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2" event={"ID":"8e95dc58-cf96-48fd-91b0-7ce6156ed573","Type":"ContainerDied","Data":"e3e6fa7439f7bcdc58bae33503ffb76ffc5e00cf45aefd32267ee44e7c1cbb7d"}
Nov 03 00:55:49.052072 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:49.052040    5012 scope.go:122] "RemoveContainer" containerID="e3e6fa7439f7bcdc58bae33503ffb76ffc5e00cf45aefd32267ee44e7c1cbb7d"
Nov 03 00:55:49.054142 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:49.054114    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:55:49.054220 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:49.054183    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:55:49.054266 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:49.054219    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:55:50.066967 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:50.066902    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2" event={"ID":"8e95dc58-cf96-48fd-91b0-7ce6156ed573","Type":"ContainerStarted","Data":"61bc9e3fc1f70ee519b5f6fd6a231acaa578cc5e69dd2169ca66080cd07dd290"}
Nov 03 00:55:50.194396 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:50.194301    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:55:50.194747 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:50.194718    5012 patch_prober.go:29] interesting pod/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Nov 03 00:55:50.194983 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:50.194954    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2" podUID="8e95dc58-cf96-48fd-91b0-7ce6156ed573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Nov 03 00:55:51.012713 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:51.012680    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v"
Nov 03 00:55:51.203416 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:51.203380    5012 patch_prober.go:29] interesting pod/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Nov 03 00:55:51.211038 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:51.210994    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2" podUID="8e95dc58-cf96-48fd-91b0-7ce6156ed573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Nov 03 00:55:52.207934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:52.206998    5012 patch_prober.go:29] interesting pod/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Nov 03 00:55:52.207934 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:52.207062    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2" podUID="8e95dc58-cf96-48fd-91b0-7ce6156ed573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Nov 03 00:55:53.195935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:53.191234    5012 patch_prober.go:29] interesting pod/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Nov 03 00:55:53.195935 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:53.191314    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2" podUID="8e95dc58-cf96-48fd-91b0-7ce6156ed573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Nov 03 00:55:54.191307 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:54.191272    5012 patch_prober.go:29] interesting pod/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Nov 03 00:55:54.191961 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:54.191927    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2" podUID="8e95dc58-cf96-48fd-91b0-7ce6156ed573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Nov 03 00:55:55.211285 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:55.211251    5012 patch_prober.go:29] interesting pod/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Nov 03 00:55:55.211683 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:55.211311    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2" podUID="8e95dc58-cf96-48fd-91b0-7ce6156ed573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Nov 03 00:55:56.195069 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:56.195038    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:55:56.195533 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:56.195507    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:55:56.196154 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:55:56.196137    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 00:56:07.369216 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:56:07.369175    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:56:08.601318 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.601284    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"]
Nov 03 00:56:08.606745 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.606720    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 00:56:08.620872 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.620850    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-726205\"/\"kube-root-ca.crt\""
Nov 03 00:56:08.631075 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.631044    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-726205\"/\"openshift-service-ca.crt\""
Nov 03 00:56:08.632444 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.632418    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-726205\"/\"test-model-726205-kserve-self-signed-certs\""
Nov 03 00:56:08.632630 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.632534    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-726205\"/\"default-dockercfg-wzzs8\""
Nov 03 00:56:08.640921 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.640876    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"]
Nov 03 00:56:08.735199 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.735172    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-tmp-dir\") pod \"test-model-726205-kserve-7fb55cb5ff-trlf8\" (UID: \"31151eab-95c9-4158-b86b-5708cf5c7329\") " pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 00:56:08.735321 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.735227    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-model-cache\") pod \"test-model-726205-kserve-7fb55cb5ff-trlf8\" (UID: \"31151eab-95c9-4158-b86b-5708cf5c7329\") " pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 00:56:08.735400 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.735378    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-dshm\") pod \"test-model-726205-kserve-7fb55cb5ff-trlf8\" (UID: \"31151eab-95c9-4158-b86b-5708cf5c7329\") " pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 00:56:08.735447 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.735433    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvzrj\" (UniqueName: \"kubernetes.io/projected/31151eab-95c9-4158-b86b-5708cf5c7329-kube-api-access-bvzrj\") pod \"test-model-726205-kserve-7fb55cb5ff-trlf8\" (UID: \"31151eab-95c9-4158-b86b-5708cf5c7329\") " pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 00:56:08.735538 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.735518    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31151eab-95c9-4158-b86b-5708cf5c7329-tls-certs\") pod \"test-model-726205-kserve-7fb55cb5ff-trlf8\" (UID: \"31151eab-95c9-4158-b86b-5708cf5c7329\") " pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 00:56:08.735630 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.735615    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-home\") pod \"test-model-726205-kserve-7fb55cb5ff-trlf8\" (UID: \"31151eab-95c9-4158-b86b-5708cf5c7329\") " pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 00:56:08.837750 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.837699    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-dshm\") pod \"test-model-726205-kserve-7fb55cb5ff-trlf8\" (UID: \"31151eab-95c9-4158-b86b-5708cf5c7329\") " pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 00:56:08.837967 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.837789    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvzrj\" (UniqueName: \"kubernetes.io/projected/31151eab-95c9-4158-b86b-5708cf5c7329-kube-api-access-bvzrj\") pod \"test-model-726205-kserve-7fb55cb5ff-trlf8\" (UID: \"31151eab-95c9-4158-b86b-5708cf5c7329\") " pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 00:56:08.837967 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.837823    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31151eab-95c9-4158-b86b-5708cf5c7329-tls-certs\") pod \"test-model-726205-kserve-7fb55cb5ff-trlf8\" (UID: \"31151eab-95c9-4158-b86b-5708cf5c7329\") " pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 00:56:08.837967 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.837935    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-home\") pod \"test-model-726205-kserve-7fb55cb5ff-trlf8\" (UID: \"31151eab-95c9-4158-b86b-5708cf5c7329\") " pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 00:56:08.838152 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.838070    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-tmp-dir\") pod \"test-model-726205-kserve-7fb55cb5ff-trlf8\" (UID: \"31151eab-95c9-4158-b86b-5708cf5c7329\") " pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 00:56:08.838152 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.838122    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-model-cache\") pod \"test-model-726205-kserve-7fb55cb5ff-trlf8\" (UID: \"31151eab-95c9-4158-b86b-5708cf5c7329\") " pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 00:56:08.838670 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.838644    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-model-cache\") pod \"test-model-726205-kserve-7fb55cb5ff-trlf8\" (UID: \"31151eab-95c9-4158-b86b-5708cf5c7329\") " pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 00:56:08.840066 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.840042    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-home\") pod \"test-model-726205-kserve-7fb55cb5ff-trlf8\" (UID: \"31151eab-95c9-4158-b86b-5708cf5c7329\") " pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 00:56:08.840342 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.840318    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-tmp-dir\") pod \"test-model-726205-kserve-7fb55cb5ff-trlf8\" (UID: \"31151eab-95c9-4158-b86b-5708cf5c7329\") " pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 00:56:08.865069 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.864588    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-dshm\") pod \"test-model-726205-kserve-7fb55cb5ff-trlf8\" (UID: \"31151eab-95c9-4158-b86b-5708cf5c7329\") " pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 00:56:08.913930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.873589    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31151eab-95c9-4158-b86b-5708cf5c7329-tls-certs\") pod \"test-model-726205-kserve-7fb55cb5ff-trlf8\" (UID: \"31151eab-95c9-4158-b86b-5708cf5c7329\") " pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 00:56:08.913930 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.874335    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvzrj\" (UniqueName: \"kubernetes.io/projected/31151eab-95c9-4158-b86b-5708cf5c7329-kube-api-access-bvzrj\") pod \"test-model-726205-kserve-7fb55cb5ff-trlf8\" (UID: \"31151eab-95c9-4158-b86b-5708cf5c7329\") " pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 00:56:08.945932 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:08.933616    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 00:56:09.307502 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:56:09.307475    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31151eab_95c9_4158_b86b_5708cf5c7329.slice/crio-c2db4563d168d24f18b5d82cb07a81b9195ceb7062168181a21c2a197f4e47c1 WatchSource:0}: Error finding container c2db4563d168d24f18b5d82cb07a81b9195ceb7062168181a21c2a197f4e47c1: Status 404 returned error can't find the container with id c2db4563d168d24f18b5d82cb07a81b9195ceb7062168181a21c2a197f4e47c1
Nov 03 00:56:09.309261 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:09.309237    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"]
Nov 03 00:56:09.402132 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:09.402105    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8" event={"ID":"31151eab-95c9-4158-b86b-5708cf5c7329","Type":"ContainerStarted","Data":"b5599386603a814ebb5d579e9c002a3b17ad3f947018c387c563139e356a4d8f"}
Nov 03 00:56:09.402260 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:09.402145    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8" event={"ID":"31151eab-95c9-4158-b86b-5708cf5c7329","Type":"ContainerStarted","Data":"c2db4563d168d24f18b5d82cb07a81b9195ceb7062168181a21c2a197f4e47c1"}
Nov 03 00:56:09.439766 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:09.439718    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8" podStartSLOduration=1.439703986 podStartE2EDuration="1.439703986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:09.438824851 +0000 UTC m=+3052.318408895" watchObservedRunningTime="2026-11-03 00:56:09.439703986 +0000 UTC m=+3052.319288029"
Nov 03 00:56:09.941111 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:09.941067    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 00:56:18.934601 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:18.934566    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 00:56:18.947863 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:18.947836    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 00:56:19.499672 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:19.499635    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 00:56:21.677704 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:21.677664    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7"]
Nov 03 00:56:21.682695 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:21.682676    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7"
Nov 03 00:56:21.700819 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:21.700795    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7"]
Nov 03 00:56:21.710850 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:21.710825    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-49d7e5-serving-cert\""
Nov 03 00:56:21.769733 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:21.769698    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7\" (UID: \"77f5e43c-70c1-4f1b-ac62-412c0939ecbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7"
Nov 03 00:56:21.769879 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:21.769836    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-maas-api-tls\") pod \"maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7\" (UID: \"77f5e43c-70c1-4f1b-ac62-412c0939ecbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7"
Nov 03 00:56:21.769963 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:21.769896    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-metrics-certs\") pod \"maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7\" (UID: \"77f5e43c-70c1-4f1b-ac62-412c0939ecbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7"
Nov 03 00:56:21.769963 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:21.769926    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxk99\" (UniqueName: \"kubernetes.io/projected/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-kube-api-access-kxk99\") pod \"maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7\" (UID: \"77f5e43c-70c1-4f1b-ac62-412c0939ecbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7"
Nov 03 00:56:21.877854 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:21.877817    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7\" (UID: \"77f5e43c-70c1-4f1b-ac62-412c0939ecbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7"
Nov 03 00:56:21.878259 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:21.878222    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-maas-api-tls\") pod \"maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7\" (UID: \"77f5e43c-70c1-4f1b-ac62-412c0939ecbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7"
Nov 03 00:56:21.878397 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:21.878294    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-metrics-certs\") pod \"maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7\" (UID: \"77f5e43c-70c1-4f1b-ac62-412c0939ecbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7"
Nov 03 00:56:21.878397 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:21.878320    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxk99\" (UniqueName: \"kubernetes.io/projected/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-kube-api-access-kxk99\") pod \"maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7\" (UID: \"77f5e43c-70c1-4f1b-ac62-412c0939ecbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7"
Nov 03 00:56:21.881047 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:21.881017    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7\" (UID: \"77f5e43c-70c1-4f1b-ac62-412c0939ecbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7"
Nov 03 00:56:21.911957 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:21.911296    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-maas-api-tls\") pod \"maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7\" (UID: \"77f5e43c-70c1-4f1b-ac62-412c0939ecbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7"
Nov 03 00:56:21.919840 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:21.915357    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-metrics-certs\") pod \"maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7\" (UID: \"77f5e43c-70c1-4f1b-ac62-412c0939ecbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7"
Nov 03 00:56:21.942339 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:21.942261    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxk99\" (UniqueName: \"kubernetes.io/projected/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-kube-api-access-kxk99\") pod \"maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7\" (UID: \"77f5e43c-70c1-4f1b-ac62-412c0939ecbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7"
Nov 03 00:56:22.002621 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:22.002584    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7"
Nov 03 00:56:22.310353 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:56:22.310325    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f5e43c_70c1_4f1b_ac62_412c0939ecbc.slice/crio-20f34d2628022a653b00d29f186b0de7d3c44076cbbd3cb30e48f4116a5458f0 WatchSource:0}: Error finding container 20f34d2628022a653b00d29f186b0de7d3c44076cbbd3cb30e48f4116a5458f0: Status 404 returned error can't find the container with id 20f34d2628022a653b00d29f186b0de7d3c44076cbbd3cb30e48f4116a5458f0
Nov 03 00:56:22.310614 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:22.310590    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7"]
Nov 03 00:56:22.519734 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:22.519697    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7" event={"ID":"77f5e43c-70c1-4f1b-ac62-412c0939ecbc","Type":"ContainerStarted","Data":"20f34d2628022a653b00d29f186b0de7d3c44076cbbd3cb30e48f4116a5458f0"}
Nov 03 00:56:23.530535 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:23.530494    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7" event={"ID":"77f5e43c-70c1-4f1b-ac62-412c0939ecbc","Type":"ContainerStarted","Data":"3a48527205c0f9c13e9d7eb7f6b0307d0e07e3883841f60b7a465938670e14c8"}
Nov 03 00:56:23.530942 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:23.530589    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7"
Nov 03 00:56:23.590315 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:23.590255    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7" podStartSLOduration=2.200567195 podStartE2EDuration="2.590238892s" totalImagesPullingTime="389.671697ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:23.583260256 +0000 UTC m=+3066.462844300" watchObservedRunningTime="2026-11-03 00:56:23.590238892 +0000 UTC m=+3066.469822935"
Nov 03 00:56:29.545208 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:56:29.545174    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7"
Nov 03 00:56:37.386555 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:56:37.386508    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:57:04.159998 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:57:04.158599    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-0f730e/e2e-usage-sim-deff0d-7b94db74b-ndvxm"]
Nov 03 00:57:04.159998 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:57:04.158832    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-0f730e/e2e-usage-sim-deff0d-7b94db74b-ndvxm" podUID="1b79e717-6fed-4b89-9d64-783211f4f037" containerName="sim" containerID="cri-o://2901a8c50fec96260010476a7713bf40bc913c97649401eb96689fdcfe1427aa" gracePeriod=30
Nov 03 00:57:04.725918 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:57:04.725881    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-0f730e/e2e-usage-sim-deff0d-7b94db74b-ndvxm"
Nov 03 00:57:04.921933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:57:04.921544    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b79e717-6fed-4b89-9d64-783211f4f037-tls\" (UniqueName: \"kubernetes.io/secret/1b79e717-6fed-4b89-9d64-783211f4f037-tls\") pod \"1b79e717-6fed-4b89-9d64-783211f4f037\" (UID: \"1b79e717-6fed-4b89-9d64-783211f4f037\") "
Nov 03 00:57:04.921933 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:57:04.921668    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b79e717-6fed-4b89-9d64-783211f4f037-kube-api-access-thwpt\" (UniqueName: \"kubernetes.io/projected/1b79e717-6fed-4b89-9d64-783211f4f037-kube-api-access-thwpt\") pod \"1b79e717-6fed-4b89-9d64-783211f4f037\" (UID: \"1b79e717-6fed-4b89-9d64-783211f4f037\") "
Nov 03 00:57:05.057323 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:57:05.055149    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b79e717-6fed-4b89-9d64-783211f4f037-kube-api-access-thwpt" pod "1b79e717-6fed-4b89-9d64-783211f4f037" (UID: "1b79e717-6fed-4b89-9d64-783211f4f037"). InnerVolumeSpecName "kube-api-access-thwpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:05.057323 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:57:05.055241    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b79e717-6fed-4b89-9d64-783211f4f037-tls" pod "1b79e717-6fed-4b89-9d64-783211f4f037" (UID: "1b79e717-6fed-4b89-9d64-783211f4f037"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:57:05.079808 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:57:05.079766    5012 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1b79e717-6fed-4b89-9d64-783211f4f037-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:57:05.079808 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:57:05.079804    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thwpt\" (UniqueName: \"kubernetes.io/projected/1b79e717-6fed-4b89-9d64-783211f4f037-kube-api-access-thwpt\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 00:57:05.161377 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:57:05.161341    5012 generic.go:356] "Generic (PLEG): container finished" podID="1b79e717-6fed-4b89-9d64-783211f4f037" containerID="2901a8c50fec96260010476a7713bf40bc913c97649401eb96689fdcfe1427aa" exitCode=0
Nov 03 00:57:05.161755 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:57:05.161392    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-0f730e/e2e-usage-sim-deff0d-7b94db74b-ndvxm"
Nov 03 00:57:05.161755 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:57:05.161425    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-0f730e/e2e-usage-sim-deff0d-7b94db74b-ndvxm" event={"ID":"1b79e717-6fed-4b89-9d64-783211f4f037","Type":"ContainerDied","Data":"2901a8c50fec96260010476a7713bf40bc913c97649401eb96689fdcfe1427aa"}
Nov 03 00:57:05.161755 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:57:05.161477    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-0f730e/e2e-usage-sim-deff0d-7b94db74b-ndvxm" event={"ID":"1b79e717-6fed-4b89-9d64-783211f4f037","Type":"ContainerDied","Data":"4ce44c88d89d53c1172b0597a6815eb488a301798e5e57b4a04bc7115437f0f8"}
Nov 03 00:57:05.161755 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:57:05.161499    5012 scope.go:122] "RemoveContainer" containerID="2901a8c50fec96260010476a7713bf40bc913c97649401eb96689fdcfe1427aa"
Nov 03 00:57:05.173453 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:57:05.173429    5012 scope.go:122] "RemoveContainer" containerID="2901a8c50fec96260010476a7713bf40bc913c97649401eb96689fdcfe1427aa"
Nov 03 00:57:05.173705 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:57:05.173686    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2901a8c50fec96260010476a7713bf40bc913c97649401eb96689fdcfe1427aa\": container with ID starting with 2901a8c50fec96260010476a7713bf40bc913c97649401eb96689fdcfe1427aa not found: ID does not exist" containerID="2901a8c50fec96260010476a7713bf40bc913c97649401eb96689fdcfe1427aa"
Nov 03 00:57:05.173761 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:57:05.173715    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2901a8c50fec96260010476a7713bf40bc913c97649401eb96689fdcfe1427aa"} err="failed to get container status \"2901a8c50fec96260010476a7713bf40bc913c97649401eb96689fdcfe1427aa\": rpc error: code = NotFound desc = could not find container \"2901a8c50fec96260010476a7713bf40bc913c97649401eb96689fdcfe1427aa\": container with ID starting with 2901a8c50fec96260010476a7713bf40bc913c97649401eb96689fdcfe1427aa not found: ID does not exist"
Nov 03 00:57:05.208714 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:57:05.208687    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-0f730e/e2e-usage-sim-deff0d-7b94db74b-ndvxm"]
Nov 03 00:57:05.219795 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:57:05.219315    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-0f730e/e2e-usage-sim-deff0d-7b94db74b-ndvxm"]
Nov 03 00:57:05.343981 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:57:05.343452    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b79e717-6fed-4b89-9d64-783211f4f037" path="/var/lib/kubelet/pods/1b79e717-6fed-4b89-9d64-783211f4f037/volumes"
Nov 03 00:57:07.364776 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:57:07.364742    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:57:37.366158 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:57:37.366117    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:58:07.412125 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:58:07.412085    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:58:19.113206 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:19.113171    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd"]
Nov 03 00:58:19.114440 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:19.114414    5012 cpu_manager.go:414] "containerMap: removing container" podUID="1b79e717-6fed-4b89-9d64-783211f4f037" containerName="sim"
Nov 03 00:58:19.114440 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:19.114438    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b79e717-6fed-4b89-9d64-783211f4f037" containerName="sim"
Nov 03 00:58:19.114658 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:19.114569    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b79e717-6fed-4b89-9d64-783211f4f037" containerName="sim"
Nov 03 00:58:19.118493 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:19.118474    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd"
Nov 03 00:58:19.137218 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:19.137188    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd"]
Nov 03 00:58:19.141450 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:19.141426    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-7f20f7-serving-cert\""
Nov 03 00:58:19.292551 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:19.292510    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs7cl\" (UniqueName: \"kubernetes.io/projected/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-kube-api-access-fs7cl\") pod \"maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd\" (UID: \"8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd"
Nov 03 00:58:19.292723 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:19.292613    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd\" (UID: \"8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd"
Nov 03 00:58:19.292843 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:19.292812    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-maas-api-tls\") pod \"maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd\" (UID: \"8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd"
Nov 03 00:58:19.292990 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:19.292968    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-metrics-certs\") pod \"maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd\" (UID: \"8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd"
Nov 03 00:58:19.394747 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:19.394656    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs7cl\" (UniqueName: \"kubernetes.io/projected/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-kube-api-access-fs7cl\") pod \"maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd\" (UID: \"8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd"
Nov 03 00:58:19.395098 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:19.395071    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd\" (UID: \"8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd"
Nov 03 00:58:19.395255 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:19.395230    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-maas-api-tls\") pod \"maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd\" (UID: \"8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd"
Nov 03 00:58:19.395395 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:19.395347    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-metrics-certs\") pod \"maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd\" (UID: \"8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd"
Nov 03 00:58:19.396247 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:19.396220    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd\" (UID: \"8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd"
Nov 03 00:58:19.398896 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:19.398873    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-metrics-certs\") pod \"maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd\" (UID: \"8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd"
Nov 03 00:58:19.399035 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:19.399013    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-maas-api-tls\") pod \"maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd\" (UID: \"8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd"
Nov 03 00:58:19.411768 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:19.411735    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs7cl\" (UniqueName: \"kubernetes.io/projected/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-kube-api-access-fs7cl\") pod \"maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd\" (UID: \"8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd"
Nov 03 00:58:19.433954 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:19.433898    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd"
Nov 03 00:58:19.746895 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:58:19.746863    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f69a68b_77af_47ad_b4d5_bcd2cc02c1ed.slice/crio-278e2deb5cd3077f2e4eec005d60b22614d814dba695ddbcce1c420e5c0bae82 WatchSource:0}: Error finding container 278e2deb5cd3077f2e4eec005d60b22614d814dba695ddbcce1c420e5c0bae82: Status 404 returned error can't find the container with id 278e2deb5cd3077f2e4eec005d60b22614d814dba695ddbcce1c420e5c0bae82
Nov 03 00:58:19.747056 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:19.747036    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd"]
Nov 03 00:58:20.243068 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:20.243040    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd" event={"ID":"8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed","Type":"ContainerStarted","Data":"842cb3f77adb8df1b2e0fdd10d1467a10e247f7a985e43fb2c7c0f9c7ac7e77e"}
Nov 03 00:58:20.243401 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:20.243095    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd" event={"ID":"8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed","Type":"ContainerStarted","Data":"278e2deb5cd3077f2e4eec005d60b22614d814dba695ddbcce1c420e5c0bae82"}
Nov 03 00:58:20.243401 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:20.243146    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd"
Nov 03 00:58:20.320518 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:20.318812    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd" podStartSLOduration=0.920579823 podStartE2EDuration="1.318791638s" totalImagesPullingTime="398.211815ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:20.297742935 +0000 UTC m=+3183.177327014" watchObservedRunningTime="2026-11-03 00:58:20.318791638 +0000 UTC m=+3183.198375684"
Nov 03 00:58:26.257313 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:26.257279    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd"
Nov 03 00:58:34.270120 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:34.270037    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs"]
Nov 03 00:58:34.277629 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:34.277602    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs"
Nov 03 00:58:34.292619 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:34.292592    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs"]
Nov 03 00:58:34.297123 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:34.297074    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-29d09b-serving-cert\""
Nov 03 00:58:34.388030 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:34.387991    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b72694be-b9a8-45df-8085-c3e958c3bf31-openshift-service-ca\") pod \"maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs\" (UID: \"b72694be-b9a8-45df-8085-c3e958c3bf31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs"
Nov 03 00:58:34.388202 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:34.388053    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b72694be-b9a8-45df-8085-c3e958c3bf31-maas-api-tls\") pod \"maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs\" (UID: \"b72694be-b9a8-45df-8085-c3e958c3bf31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs"
Nov 03 00:58:34.388202 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:34.388182    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b72694be-b9a8-45df-8085-c3e958c3bf31-metrics-certs\") pod \"maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs\" (UID: \"b72694be-b9a8-45df-8085-c3e958c3bf31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs"
Nov 03 00:58:34.388320 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:34.388213    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5dsd\" (UniqueName: \"kubernetes.io/projected/b72694be-b9a8-45df-8085-c3e958c3bf31-kube-api-access-j5dsd\") pod \"maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs\" (UID: \"b72694be-b9a8-45df-8085-c3e958c3bf31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs"
Nov 03 00:58:34.489661 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:34.489630    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b72694be-b9a8-45df-8085-c3e958c3bf31-openshift-service-ca\") pod \"maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs\" (UID: \"b72694be-b9a8-45df-8085-c3e958c3bf31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs"
Nov 03 00:58:34.489844 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:34.489683    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b72694be-b9a8-45df-8085-c3e958c3bf31-maas-api-tls\") pod \"maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs\" (UID: \"b72694be-b9a8-45df-8085-c3e958c3bf31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs"
Nov 03 00:58:34.489844 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:34.489749    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b72694be-b9a8-45df-8085-c3e958c3bf31-metrics-certs\") pod \"maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs\" (UID: \"b72694be-b9a8-45df-8085-c3e958c3bf31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs"
Nov 03 00:58:34.489844 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:34.489769    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5dsd\" (UniqueName: \"kubernetes.io/projected/b72694be-b9a8-45df-8085-c3e958c3bf31-kube-api-access-j5dsd\") pod \"maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs\" (UID: \"b72694be-b9a8-45df-8085-c3e958c3bf31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs"
Nov 03 00:58:34.490398 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:34.490370    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b72694be-b9a8-45df-8085-c3e958c3bf31-openshift-service-ca\") pod \"maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs\" (UID: \"b72694be-b9a8-45df-8085-c3e958c3bf31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs"
Nov 03 00:58:34.492424 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:34.492395    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b72694be-b9a8-45df-8085-c3e958c3bf31-maas-api-tls\") pod \"maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs\" (UID: \"b72694be-b9a8-45df-8085-c3e958c3bf31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs"
Nov 03 00:58:34.492537 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:34.492395    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b72694be-b9a8-45df-8085-c3e958c3bf31-metrics-certs\") pod \"maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs\" (UID: \"b72694be-b9a8-45df-8085-c3e958c3bf31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs"
Nov 03 00:58:34.506683 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:34.506656    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5dsd\" (UniqueName: \"kubernetes.io/projected/b72694be-b9a8-45df-8085-c3e958c3bf31-kube-api-access-j5dsd\") pod \"maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs\" (UID: \"b72694be-b9a8-45df-8085-c3e958c3bf31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs"
Nov 03 00:58:34.604359 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:34.604327    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs"
Nov 03 00:58:34.942363 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:58:34.942334    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72694be_b9a8_45df_8085_c3e958c3bf31.slice/crio-b0506e34fe58e65fb996b0d2515d0149ed3940fc19bb79f4b393eed04031b711 WatchSource:0}: Error finding container b0506e34fe58e65fb996b0d2515d0149ed3940fc19bb79f4b393eed04031b711: Status 404 returned error can't find the container with id b0506e34fe58e65fb996b0d2515d0149ed3940fc19bb79f4b393eed04031b711
Nov 03 00:58:34.944344 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:34.944321    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs"]
Nov 03 00:58:35.399975 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:35.399945    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs" event={"ID":"b72694be-b9a8-45df-8085-c3e958c3bf31","Type":"ContainerStarted","Data":"5f333bede3c36f306268e5de9493b6df8cad10e5da70fdae7ff10f1105633a95"}
Nov 03 00:58:35.400383 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:35.400002    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs" event={"ID":"b72694be-b9a8-45df-8085-c3e958c3bf31","Type":"ContainerStarted","Data":"b0506e34fe58e65fb996b0d2515d0149ed3940fc19bb79f4b393eed04031b711"}
Nov 03 00:58:35.400383 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:35.400092    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs"
Nov 03 00:58:35.437114 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:35.437070    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs" podStartSLOduration=1.090861439 podStartE2EDuration="1.437056535s" totalImagesPullingTime="346.195096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:35.434950462 +0000 UTC m=+3198.314534502" watchObservedRunningTime="2026-11-03 00:58:35.437056535 +0000 UTC m=+3198.316640578"
Nov 03 00:58:37.451097 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:58:37.451052    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:58:39.676122 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:39.676088    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"]
Nov 03 00:58:39.681583 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:39.681555    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 00:58:39.700148 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:39.700121    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"]
Nov 03 00:58:39.702299 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:39.702277    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-883a9e\"/\"kube-root-ca.crt\""
Nov 03 00:58:39.702299 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:39.702285    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-883a9e\"/\"openshift-service-ca.crt\""
Nov 03 00:58:39.704789 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:39.704766    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-883a9e\"/\"test-model-883a9e-kserve-self-signed-certs\""
Nov 03 00:58:39.704789 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:39.704775    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-883a9e\"/\"default-dockercfg-lx6tx\""
Nov 03 00:58:39.825812 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:39.825775    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hss7\" (UniqueName: \"kubernetes.io/projected/209e0827-a624-4090-bcc4-4ac72c7eb1e6-kube-api-access-2hss7\") pod \"test-model-883a9e-kserve-86d6f745dc-xs7fj\" (UID: \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\") " pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 00:58:39.826029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:39.825852    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-home\") pod \"test-model-883a9e-kserve-86d6f745dc-xs7fj\" (UID: \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\") " pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 00:58:39.826029 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:39.825948    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-tmp-dir\") pod \"test-model-883a9e-kserve-86d6f745dc-xs7fj\" (UID: \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\") " pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 00:58:39.826157 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:39.826077    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-model-cache\") pod \"test-model-883a9e-kserve-86d6f745dc-xs7fj\" (UID: \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\") " pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 00:58:39.826230 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:39.826205    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/209e0827-a624-4090-bcc4-4ac72c7eb1e6-tls-certs\") pod \"test-model-883a9e-kserve-86d6f745dc-xs7fj\" (UID: \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\") " pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 00:58:39.826339 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:39.826316    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-dshm\") pod \"test-model-883a9e-kserve-86d6f745dc-xs7fj\" (UID: \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\") " pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 00:58:39.936028 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:39.931731    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hss7\" (UniqueName: \"kubernetes.io/projected/209e0827-a624-4090-bcc4-4ac72c7eb1e6-kube-api-access-2hss7\") pod \"test-model-883a9e-kserve-86d6f745dc-xs7fj\" (UID: \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\") " pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 00:58:39.936028 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:39.931781    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-home\") pod \"test-model-883a9e-kserve-86d6f745dc-xs7fj\" (UID: \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\") " pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 00:58:39.936028 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:39.931819    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-tmp-dir\") pod \"test-model-883a9e-kserve-86d6f745dc-xs7fj\" (UID: \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\") " pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 00:58:39.936028 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:39.931970    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-model-cache\") pod \"test-model-883a9e-kserve-86d6f745dc-xs7fj\" (UID: \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\") " pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 00:58:39.936028 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:39.932108    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/209e0827-a624-4090-bcc4-4ac72c7eb1e6-tls-certs\") pod \"test-model-883a9e-kserve-86d6f745dc-xs7fj\" (UID: \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\") " pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 00:58:39.936028 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:39.932161    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-dshm\") pod \"test-model-883a9e-kserve-86d6f745dc-xs7fj\" (UID: \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\") " pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 00:58:39.936028 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:39.932885    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-tmp-dir\") pod \"test-model-883a9e-kserve-86d6f745dc-xs7fj\" (UID: \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\") " pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 00:58:39.936028 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:39.933678    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-home\") pod \"test-model-883a9e-kserve-86d6f745dc-xs7fj\" (UID: \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\") " pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 00:58:39.936028 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:39.934542    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-model-cache\") pod \"test-model-883a9e-kserve-86d6f745dc-xs7fj\" (UID: \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\") " pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 00:58:39.976043 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:39.976005    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-dshm\") pod \"test-model-883a9e-kserve-86d6f745dc-xs7fj\" (UID: \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\") " pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 00:58:39.999936 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:39.992456    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/209e0827-a624-4090-bcc4-4ac72c7eb1e6-tls-certs\") pod \"test-model-883a9e-kserve-86d6f745dc-xs7fj\" (UID: \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\") " pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 00:58:40.016771 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:40.016721    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hss7\" (UniqueName: \"kubernetes.io/projected/209e0827-a624-4090-bcc4-4ac72c7eb1e6-kube-api-access-2hss7\") pod \"test-model-883a9e-kserve-86d6f745dc-xs7fj\" (UID: \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\") " pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 00:58:40.018200 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:40.018170    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 00:58:40.399672 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:58:40.399637    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod209e0827_a624_4090_bcc4_4ac72c7eb1e6.slice/crio-e743a2e3c4ab318badd43b68eea586da61081559e318833751372a78a31aefa0 WatchSource:0}: Error finding container e743a2e3c4ab318badd43b68eea586da61081559e318833751372a78a31aefa0: Status 404 returned error can't find the container with id e743a2e3c4ab318badd43b68eea586da61081559e318833751372a78a31aefa0
Nov 03 00:58:40.400154 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:40.400127    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"]
Nov 03 00:58:40.494479 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:40.494282    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj" event={"ID":"209e0827-a624-4090-bcc4-4ac72c7eb1e6","Type":"ContainerStarted","Data":"23715d6872d34e2e403afd43e060ba71fbdbdf35355c0b597b3b69e11dceb7d2"}
Nov 03 00:58:40.494479 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:40.494340    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj" event={"ID":"209e0827-a624-4090-bcc4-4ac72c7eb1e6","Type":"ContainerStarted","Data":"e743a2e3c4ab318badd43b68eea586da61081559e318833751372a78a31aefa0"}
Nov 03 00:58:40.530789 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:40.530744    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj" podStartSLOduration=1.530730491 podStartE2EDuration="1.530730491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:40.529504384 +0000 UTC m=+3203.409088427" watchObservedRunningTime="2026-11-03 00:58:40.530730491 +0000 UTC m=+3203.410314556"
Nov 03 00:58:41.022393 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:41.022361    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 00:58:41.430375 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:41.430346    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs"
Nov 03 00:58:50.021869 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:50.021831    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 00:58:50.035319 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:50.035293    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 00:58:50.600500 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:58:50.600466    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 00:59:07.367035 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:59:07.366997    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:59:37.366036 ip-10-0-133-161 kubenswrapper[5012]: E1103 00:59:37.365994    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 00:59:48.433928 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.427402    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"]
Nov 03 00:59:48.441518 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.441491    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"]
Nov 03 00:59:48.441980 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.441631    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 00:59:48.466375 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.466350    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7f20f7\"/\"kube-root-ca.crt\""
Nov 03 00:59:48.466507 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.466391    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7f20f7\"/\"default-dockercfg-lf67w\""
Nov 03 00:59:48.466507 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.466401    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7f20f7\"/\"openshift-service-ca.crt\""
Nov 03 00:59:48.466507 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.466428    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7f20f7\"/\"test-model-7f20f7-kserve-self-signed-certs\""
Nov 03 00:59:48.529550 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.529516    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72d32ad0-f7d6-4c17-8781-83d65335f0e6-tls-certs\") pod \"test-model-7f20f7-kserve-f656db587-plwfp\" (UID: \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\") " pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 00:59:48.529691 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.529567    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7sw7\" (UniqueName: \"kubernetes.io/projected/72d32ad0-f7d6-4c17-8781-83d65335f0e6-kube-api-access-h7sw7\") pod \"test-model-7f20f7-kserve-f656db587-plwfp\" (UID: \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\") " pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 00:59:48.529691 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.529669    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-model-cache\") pod \"test-model-7f20f7-kserve-f656db587-plwfp\" (UID: \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\") " pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 00:59:48.529903 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.529888    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-dshm\") pod \"test-model-7f20f7-kserve-f656db587-plwfp\" (UID: \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\") " pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 00:59:48.529992 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.529976    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-tmp-dir\") pod \"test-model-7f20f7-kserve-f656db587-plwfp\" (UID: \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\") " pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 00:59:48.530033 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.530012    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-home\") pod \"test-model-7f20f7-kserve-f656db587-plwfp\" (UID: \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\") " pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 00:59:48.631172 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.631138    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-model-cache\") pod \"test-model-7f20f7-kserve-f656db587-plwfp\" (UID: \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\") " pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 00:59:48.631340 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.631315    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-dshm\") pod \"test-model-7f20f7-kserve-f656db587-plwfp\" (UID: \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\") " pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 00:59:48.631388 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.631363    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-tmp-dir\") pod \"test-model-7f20f7-kserve-f656db587-plwfp\" (UID: \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\") " pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 00:59:48.631434 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.631393    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-home\") pod \"test-model-7f20f7-kserve-f656db587-plwfp\" (UID: \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\") " pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 00:59:48.631434 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.631427    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72d32ad0-f7d6-4c17-8781-83d65335f0e6-tls-certs\") pod \"test-model-7f20f7-kserve-f656db587-plwfp\" (UID: \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\") " pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 00:59:48.631537 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.631457    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7sw7\" (UniqueName: \"kubernetes.io/projected/72d32ad0-f7d6-4c17-8781-83d65335f0e6-kube-api-access-h7sw7\") pod \"test-model-7f20f7-kserve-f656db587-plwfp\" (UID: \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\") " pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 00:59:48.631537 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.631516    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-model-cache\") pod \"test-model-7f20f7-kserve-f656db587-plwfp\" (UID: \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\") " pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 00:59:48.631760 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.631740    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-tmp-dir\") pod \"test-model-7f20f7-kserve-f656db587-plwfp\" (UID: \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\") " pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 00:59:48.631845 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.631825    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-home\") pod \"test-model-7f20f7-kserve-f656db587-plwfp\" (UID: \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\") " pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 00:59:48.633507 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.633482    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-dshm\") pod \"test-model-7f20f7-kserve-f656db587-plwfp\" (UID: \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\") " pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 00:59:48.633886 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.633866    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72d32ad0-f7d6-4c17-8781-83d65335f0e6-tls-certs\") pod \"test-model-7f20f7-kserve-f656db587-plwfp\" (UID: \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\") " pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 00:59:48.695007 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.694928    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7sw7\" (UniqueName: \"kubernetes.io/projected/72d32ad0-f7d6-4c17-8781-83d65335f0e6-kube-api-access-h7sw7\") pod \"test-model-7f20f7-kserve-f656db587-plwfp\" (UID: \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\") " pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 00:59:48.764509 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:48.764474    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 00:59:49.105311 ip-10-0-133-161 kubenswrapper[5012]: W1103 00:59:49.105245    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72d32ad0_f7d6_4c17_8781_83d65335f0e6.slice/crio-f8c07a8bc0cb6eebc1366f2c20189277ce83e5cf64ab85109a49a877fdd83050 WatchSource:0}: Error finding container f8c07a8bc0cb6eebc1366f2c20189277ce83e5cf64ab85109a49a877fdd83050: Status 404 returned error can't find the container with id f8c07a8bc0cb6eebc1366f2c20189277ce83e5cf64ab85109a49a877fdd83050
Nov 03 00:59:49.105799 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:49.105729    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"]
Nov 03 00:59:49.289691 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:49.289650    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp" event={"ID":"72d32ad0-f7d6-4c17-8781-83d65335f0e6","Type":"ContainerStarted","Data":"930139bcac57df84bddd7baf280b712e14ea28daf3a48cdbaa01e10b7d8b1132"}
Nov 03 00:59:49.289691 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:49.289704    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp" event={"ID":"72d32ad0-f7d6-4c17-8781-83d65335f0e6","Type":"ContainerStarted","Data":"f8c07a8bc0cb6eebc1366f2c20189277ce83e5cf64ab85109a49a877fdd83050"}
Nov 03 00:59:49.329865 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:49.329816    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp" podStartSLOduration=1.329801858 podStartE2EDuration="1.329801858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:49.324268978 +0000 UTC m=+3272.203853022" watchObservedRunningTime="2026-11-03 00:59:49.329801858 +0000 UTC m=+3272.209385929"
Nov 03 00:59:49.774896 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:49.770997    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 00:59:58.765897 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:58.765816    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 00:59:58.778537 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:58.778512    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 00:59:59.387279 ip-10-0-133-161 kubenswrapper[5012]: I1103 00:59:59.387247    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 01:00:00.364969 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.364931    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw"]
Nov 03 01:00:00.376295 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.376263    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw"
Nov 03 01:00:00.382730 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.382694    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2"]
Nov 03 01:00:00.389653 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.389627    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2"
Nov 03 01:00:00.398621 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.398589    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw"]
Nov 03 01:00:00.410414 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.410388    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv"]
Nov 03 01:00:00.417649 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.417610    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv"
Nov 03 01:00:00.444425 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.444403    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2"]
Nov 03 01:00:00.453868 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.453823    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:00:00.454976 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.454932    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk"]
Nov 03 01:00:00.460338 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.460317    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk"
Nov 03 01:00:00.467957 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.467870    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk"]
Nov 03 01:00:00.474181 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.474164    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv"]
Nov 03 01:00:00.476520 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.476500    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgvn2\" (UniqueName: \"kubernetes.io/projected/6b934863-00e3-4318-9d21-8bb7b64d9761-kube-api-access-xgvn2\") pod \"maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw\" (UID: \"6b934863-00e3-4318-9d21-8bb7b64d9761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw"
Nov 03 01:00:00.476594 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.476562    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8z2t\" (UniqueName: \"kubernetes.io/projected/703067f2-bbcf-475f-8ed2-7146f5ad5f6b-kube-api-access-m8z2t\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv\" (UID: \"703067f2-bbcf-475f-8ed2-7146f5ad5f6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv"
Nov 03 01:00:00.476594 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.476583    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/703067f2-bbcf-475f-8ed2-7146f5ad5f6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv\" (UID: \"703067f2-bbcf-475f-8ed2-7146f5ad5f6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv"
Nov 03 01:00:00.476699 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.476643    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eea71f4-f5d2-4530-9105-8e4e4c77c4fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2\" (UID: \"7eea71f4-f5d2-4530-9105-8e4e4c77c4fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2"
Nov 03 01:00:00.476756 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.476731    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b934863-00e3-4318-9d21-8bb7b64d9761-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw\" (UID: \"6b934863-00e3-4318-9d21-8bb7b64d9761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw"
Nov 03 01:00:00.476756 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.476753    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2k76\" (UniqueName: \"kubernetes.io/projected/7eea71f4-f5d2-4530-9105-8e4e4c77c4fd-kube-api-access-f2k76\") pod \"maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2\" (UID: \"7eea71f4-f5d2-4530-9105-8e4e4c77c4fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2"
Nov 03 01:00:00.578191 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.578157    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbp2r\" (UniqueName: \"kubernetes.io/projected/f1faca4e-fef0-4781-a61b-52d107115094-kube-api-access-wbp2r\") pod \"maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk\" (UID: \"f1faca4e-fef0-4781-a61b-52d107115094\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk"
Nov 03 01:00:00.578375 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.578199    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b934863-00e3-4318-9d21-8bb7b64d9761-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw\" (UID: \"6b934863-00e3-4318-9d21-8bb7b64d9761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw"
Nov 03 01:00:00.578375 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.578226    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2k76\" (UniqueName: \"kubernetes.io/projected/7eea71f4-f5d2-4530-9105-8e4e4c77c4fd-kube-api-access-f2k76\") pod \"maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2\" (UID: \"7eea71f4-f5d2-4530-9105-8e4e4c77c4fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2"
Nov 03 01:00:00.578375 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.578348    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgvn2\" (UniqueName: \"kubernetes.io/projected/6b934863-00e3-4318-9d21-8bb7b64d9761-kube-api-access-xgvn2\") pod \"maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw\" (UID: \"6b934863-00e3-4318-9d21-8bb7b64d9761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw"
Nov 03 01:00:00.578584 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.578400    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1faca4e-fef0-4781-a61b-52d107115094-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk\" (UID: \"f1faca4e-fef0-4781-a61b-52d107115094\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk"
Nov 03 01:00:00.578584 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.578438    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8z2t\" (UniqueName: \"kubernetes.io/projected/703067f2-bbcf-475f-8ed2-7146f5ad5f6b-kube-api-access-m8z2t\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv\" (UID: \"703067f2-bbcf-475f-8ed2-7146f5ad5f6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv"
Nov 03 01:00:00.578584 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.578459    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/703067f2-bbcf-475f-8ed2-7146f5ad5f6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv\" (UID: \"703067f2-bbcf-475f-8ed2-7146f5ad5f6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv"
Nov 03 01:00:00.578762 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.578590    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eea71f4-f5d2-4530-9105-8e4e4c77c4fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2\" (UID: \"7eea71f4-f5d2-4530-9105-8e4e4c77c4fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2"
Nov 03 01:00:00.579054 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.579032    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b934863-00e3-4318-9d21-8bb7b64d9761-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw\" (UID: \"6b934863-00e3-4318-9d21-8bb7b64d9761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw"
Nov 03 01:00:00.579152 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.579138    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/703067f2-bbcf-475f-8ed2-7146f5ad5f6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv\" (UID: \"703067f2-bbcf-475f-8ed2-7146f5ad5f6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv"
Nov 03 01:00:00.579262 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.579242    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eea71f4-f5d2-4530-9105-8e4e4c77c4fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2\" (UID: \"7eea71f4-f5d2-4530-9105-8e4e4c77c4fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2"
Nov 03 01:00:00.596274 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.596247    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8z2t\" (UniqueName: \"kubernetes.io/projected/703067f2-bbcf-475f-8ed2-7146f5ad5f6b-kube-api-access-m8z2t\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv\" (UID: \"703067f2-bbcf-475f-8ed2-7146f5ad5f6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv"
Nov 03 01:00:00.596390 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.596320    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgvn2\" (UniqueName: \"kubernetes.io/projected/6b934863-00e3-4318-9d21-8bb7b64d9761-kube-api-access-xgvn2\") pod \"maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw\" (UID: \"6b934863-00e3-4318-9d21-8bb7b64d9761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw"
Nov 03 01:00:00.596482 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.596460    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2k76\" (UniqueName: \"kubernetes.io/projected/7eea71f4-f5d2-4530-9105-8e4e4c77c4fd-kube-api-access-f2k76\") pod \"maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2\" (UID: \"7eea71f4-f5d2-4530-9105-8e4e4c77c4fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2"
Nov 03 01:00:00.719236 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.713931    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbp2r\" (UniqueName: \"kubernetes.io/projected/f1faca4e-fef0-4781-a61b-52d107115094-kube-api-access-wbp2r\") pod \"maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk\" (UID: \"f1faca4e-fef0-4781-a61b-52d107115094\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk"
Nov 03 01:00:00.719236 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.714139    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1faca4e-fef0-4781-a61b-52d107115094-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk\" (UID: \"f1faca4e-fef0-4781-a61b-52d107115094\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk"
Nov 03 01:00:00.719236 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.715231    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1faca4e-fef0-4781-a61b-52d107115094-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk\" (UID: \"f1faca4e-fef0-4781-a61b-52d107115094\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk"
Nov 03 01:00:00.729987 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.726073    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw"
Nov 03 01:00:00.741054 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.739662    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbp2r\" (UniqueName: \"kubernetes.io/projected/f1faca4e-fef0-4781-a61b-52d107115094-kube-api-access-wbp2r\") pod \"maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk\" (UID: \"f1faca4e-fef0-4781-a61b-52d107115094\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk"
Nov 03 01:00:00.743665 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.743626    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2"
Nov 03 01:00:00.760210 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.760184    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv"
Nov 03 01:00:00.787443 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:00.787415    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk"
Nov 03 01:00:01.117402 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:01.117369    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw"]
Nov 03 01:00:01.124599 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:01.124559    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2c2816f-5dfxh"]
Nov 03 01:00:01.127735 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:00:01.127701    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b934863_00e3_4318_9d21_8bb7b64d9761.slice/crio-32c23bd0ae70fc2ee8ac90dfb6a98e18ee5508627f615534cb888031e4ecbc77 WatchSource:0}: Error finding container 32c23bd0ae70fc2ee8ac90dfb6a98e18ee5508627f615534cb888031e4ecbc77: Status 404 returned error can't find the container with id 32c23bd0ae70fc2ee8ac90dfb6a98e18ee5508627f615534cb888031e4ecbc77
Nov 03 01:00:01.132334 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:00:01.132310    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703067f2_bbcf_475f_8ed2_7146f5ad5f6b.slice/crio-4592c31e3ca138e8e0d58ffd6ccdebc03bb4ce429c26fe51088b8067bffe242c WatchSource:0}: Error finding container 4592c31e3ca138e8e0d58ffd6ccdebc03bb4ce429c26fe51088b8067bffe242c: Status 404 returned error can't find the container with id 4592c31e3ca138e8e0d58ffd6ccdebc03bb4ce429c26fe51088b8067bffe242c
Nov 03 01:00:01.133035 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:01.133011    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv"]
Nov 03 01:00:01.133134 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:01.133115    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c2816f-5dfxh"
Nov 03 01:00:01.133824 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:00:01.133742    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eea71f4_f5d2_4530_9105_8e4e4c77c4fd.slice/crio-d70cb75c0d8d69125de2d45c5a9f950d47c525b67d6d18a3112913f1a2f8e206 WatchSource:0}: Error finding container d70cb75c0d8d69125de2d45c5a9f950d47c525b67d6d18a3112913f1a2f8e206: Status 404 returned error can't find the container with id d70cb75c0d8d69125de2d45c5a9f950d47c525b67d6d18a3112913f1a2f8e206
Nov 03 01:00:01.138010 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:01.137989    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2"]
Nov 03 01:00:01.149274 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:01.145884    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:00:01.149274 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:01.146612    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2c2816f-5dfxh"]
Nov 03 01:00:01.153507 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:01.153483    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk"]
Nov 03 01:00:01.163464 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:00:01.162792    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1faca4e_fef0_4781_a61b_52d107115094.slice/crio-1cc83be1c67e6c78e47e957065b6963ac16269a7da60d84517f325636f3020c2 WatchSource:0}: Error finding container 1cc83be1c67e6c78e47e957065b6963ac16269a7da60d84517f325636f3020c2: Status 404 returned error can't find the container with id 1cc83be1c67e6c78e47e957065b6963ac16269a7da60d84517f325636f3020c2
Nov 03 01:00:01.228561 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:01.228532    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824e8c00-991b-43a9-a368-04b7baa1e746-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2c2816f-5dfxh\" (UID: \"824e8c00-991b-43a9-a368-04b7baa1e746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c2816f-5dfxh"
Nov 03 01:00:01.228691 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:01.228668    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824e8c00-991b-43a9-a368-04b7baa1e746-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2c2816f-5dfxh\" (UID: \"824e8c00-991b-43a9-a368-04b7baa1e746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c2816f-5dfxh"
Nov 03 01:00:01.228921 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:01.228887    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7965\" (UniqueName: \"kubernetes.io/projected/824e8c00-991b-43a9-a368-04b7baa1e746-kube-api-access-k7965\") pod \"maas-api-revoke-sub-c2c2816f-5dfxh\" (UID: \"824e8c00-991b-43a9-a368-04b7baa1e746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c2816f-5dfxh"
Nov 03 01:00:01.330694 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:01.330621    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824e8c00-991b-43a9-a368-04b7baa1e746-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2c2816f-5dfxh\" (UID: \"824e8c00-991b-43a9-a368-04b7baa1e746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c2816f-5dfxh"
Nov 03 01:00:01.330821 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:01.330755    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824e8c00-991b-43a9-a368-04b7baa1e746-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2c2816f-5dfxh\" (UID: \"824e8c00-991b-43a9-a368-04b7baa1e746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c2816f-5dfxh"
Nov 03 01:00:01.330976 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:01.330961    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7965\" (UniqueName: \"kubernetes.io/projected/824e8c00-991b-43a9-a368-04b7baa1e746-kube-api-access-k7965\") pod \"maas-api-revoke-sub-c2c2816f-5dfxh\" (UID: \"824e8c00-991b-43a9-a368-04b7baa1e746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c2816f-5dfxh"
Nov 03 01:00:01.331422 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:01.331400    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824e8c00-991b-43a9-a368-04b7baa1e746-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2c2816f-5dfxh\" (UID: \"824e8c00-991b-43a9-a368-04b7baa1e746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c2816f-5dfxh"
Nov 03 01:00:01.333075 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:01.333051    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824e8c00-991b-43a9-a368-04b7baa1e746-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2c2816f-5dfxh\" (UID: \"824e8c00-991b-43a9-a368-04b7baa1e746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c2816f-5dfxh"
Nov 03 01:00:01.348159 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:01.348138    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7965\" (UniqueName: \"kubernetes.io/projected/824e8c00-991b-43a9-a368-04b7baa1e746-kube-api-access-k7965\") pod \"maas-api-revoke-sub-c2c2816f-5dfxh\" (UID: \"824e8c00-991b-43a9-a368-04b7baa1e746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c2816f-5dfxh"
Nov 03 01:00:01.411748 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:01.411714    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2" event={"ID":"7eea71f4-f5d2-4530-9105-8e4e4c77c4fd","Type":"ContainerStarted","Data":"d70cb75c0d8d69125de2d45c5a9f950d47c525b67d6d18a3112913f1a2f8e206"}
Nov 03 01:00:01.413136 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:01.413106    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw" event={"ID":"6b934863-00e3-4318-9d21-8bb7b64d9761","Type":"ContainerStarted","Data":"32c23bd0ae70fc2ee8ac90dfb6a98e18ee5508627f615534cb888031e4ecbc77"}
Nov 03 01:00:01.414337 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:01.414310    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv" event={"ID":"703067f2-bbcf-475f-8ed2-7146f5ad5f6b","Type":"ContainerStarted","Data":"4592c31e3ca138e8e0d58ffd6ccdebc03bb4ce429c26fe51088b8067bffe242c"}
Nov 03 01:00:01.415499 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:01.415471    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk" event={"ID":"f1faca4e-fef0-4781-a61b-52d107115094","Type":"ContainerStarted","Data":"1cc83be1c67e6c78e47e957065b6963ac16269a7da60d84517f325636f3020c2"}
Nov 03 01:00:01.455441 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:01.455412    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c2816f-5dfxh"
Nov 03 01:00:01.766981 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:00:01.766950    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824e8c00_991b_43a9_a368_04b7baa1e746.slice/crio-d7484ca51fdcdd4e4ba1c6052669b98ceaea73a73f233f6ed4fc90610c863afb WatchSource:0}: Error finding container d7484ca51fdcdd4e4ba1c6052669b98ceaea73a73f233f6ed4fc90610c863afb: Status 404 returned error can't find the container with id d7484ca51fdcdd4e4ba1c6052669b98ceaea73a73f233f6ed4fc90610c863afb
Nov 03 01:00:01.768328 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:01.768304    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2c2816f-5dfxh"]
Nov 03 01:00:02.425893 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:02.425865    5012 generic.go:356] "Generic (PLEG): container finished" podID="7eea71f4-f5d2-4530-9105-8e4e4c77c4fd" containerID="d1fd39927190fe05a1e156ca940bf92943d45ff5ad87559b6cce545aed7fc0db" exitCode=60
Nov 03 01:00:02.426338 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:02.425944    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2" event={"ID":"7eea71f4-f5d2-4530-9105-8e4e4c77c4fd","Type":"ContainerDied","Data":"d1fd39927190fe05a1e156ca940bf92943d45ff5ad87559b6cce545aed7fc0db"}
Nov 03 01:00:02.426338 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:02.426269    5012 scope.go:122] "RemoveContainer" containerID="d1fd39927190fe05a1e156ca940bf92943d45ff5ad87559b6cce545aed7fc0db"
Nov 03 01:00:02.427802 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:02.427782    5012 generic.go:356] "Generic (PLEG): container finished" podID="6b934863-00e3-4318-9d21-8bb7b64d9761" containerID="e10cc87fd1cb077913501f84244ef3918030e8d621ca922d7e3917d934764145" exitCode=60
Nov 03 01:00:02.427939 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:02.427867    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw" event={"ID":"6b934863-00e3-4318-9d21-8bb7b64d9761","Type":"ContainerDied","Data":"e10cc87fd1cb077913501f84244ef3918030e8d621ca922d7e3917d934764145"}
Nov 03 01:00:02.428178 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:02.428138    5012 scope.go:122] "RemoveContainer" containerID="e10cc87fd1cb077913501f84244ef3918030e8d621ca922d7e3917d934764145"
Nov 03 01:00:02.430099 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:02.430080    5012 generic.go:356] "Generic (PLEG): container finished" podID="703067f2-bbcf-475f-8ed2-7146f5ad5f6b" containerID="4b4aadca9f12f2bee6da4f73aefffa084e49f022a31d9e899b65cf53b80d66a9" exitCode=60
Nov 03 01:00:02.430174 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:02.430157    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv" event={"ID":"703067f2-bbcf-475f-8ed2-7146f5ad5f6b","Type":"ContainerDied","Data":"4b4aadca9f12f2bee6da4f73aefffa084e49f022a31d9e899b65cf53b80d66a9"}
Nov 03 01:00:02.430624 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:02.430487    5012 scope.go:122] "RemoveContainer" containerID="4b4aadca9f12f2bee6da4f73aefffa084e49f022a31d9e899b65cf53b80d66a9"
Nov 03 01:00:02.432284 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:02.432201    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c2816f-5dfxh" event={"ID":"824e8c00-991b-43a9-a368-04b7baa1e746","Type":"ContainerStarted","Data":"66145bb3ea18ba9db4a1dc1730ffb3740b59ec1d2a596b39442c6a9d9d2d3662"}
Nov 03 01:00:02.432284 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:02.432247    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c2816f-5dfxh" event={"ID":"824e8c00-991b-43a9-a368-04b7baa1e746","Type":"ContainerStarted","Data":"d7484ca51fdcdd4e4ba1c6052669b98ceaea73a73f233f6ed4fc90610c863afb"}
Nov 03 01:00:02.433652 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:02.433629    5012 generic.go:356] "Generic (PLEG): container finished" podID="f1faca4e-fef0-4781-a61b-52d107115094" containerID="49c9038bc4cc7ba122ceb093e20bf7f6b00508d66241491f84d4d85e2613fae9" exitCode=60
Nov 03 01:00:02.433766 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:02.433707    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk" event={"ID":"f1faca4e-fef0-4781-a61b-52d107115094","Type":"ContainerDied","Data":"49c9038bc4cc7ba122ceb093e20bf7f6b00508d66241491f84d4d85e2613fae9"}
Nov 03 01:00:02.433954 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:02.433905    5012 scope.go:122] "RemoveContainer" containerID="49c9038bc4cc7ba122ceb093e20bf7f6b00508d66241491f84d4d85e2613fae9"
Nov 03 01:00:02.609391 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:02.609340    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c2816f-5dfxh" podStartSLOduration=1.6093259629999999 podStartE2EDuration="1.609325963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:02.604467872 +0000 UTC m=+3285.484051915" watchObservedRunningTime="2026-11-03 01:00:02.609325963 +0000 UTC m=+3285.488910005"
Nov 03 01:00:03.445895 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:03.445854    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk" event={"ID":"f1faca4e-fef0-4781-a61b-52d107115094","Type":"ContainerStarted","Data":"ac9913e205a5f6d62de91beeede1f3f890952b4c6967d20988aeef0727cca8fc"}
Nov 03 01:00:03.448211 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:03.448179    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2" event={"ID":"7eea71f4-f5d2-4530-9105-8e4e4c77c4fd","Type":"ContainerStarted","Data":"eb0b6ff3f2bd530c261b99782a2099c25337249721ac2771b885e3921431fe57"}
Nov 03 01:00:03.450507 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:03.450478    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw" event={"ID":"6b934863-00e3-4318-9d21-8bb7b64d9761","Type":"ContainerStarted","Data":"d0fecac02804c6d4699a7f60115a8fab5dbc3a78718034b8b323b6354531f970"}
Nov 03 01:00:03.452863 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:03.452823    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv" event={"ID":"703067f2-bbcf-475f-8ed2-7146f5ad5f6b","Type":"ContainerStarted","Data":"5c3cc77e87c6639fd89a04e576374c18d3a7e12e69a494bd5ee390b012a3d66c"}
Nov 03 01:00:03.455330 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:03.455297    5012 generic.go:356] "Generic (PLEG): container finished" podID="824e8c00-991b-43a9-a368-04b7baa1e746" containerID="66145bb3ea18ba9db4a1dc1730ffb3740b59ec1d2a596b39442c6a9d9d2d3662" exitCode=0
Nov 03 01:00:03.455495 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:03.455347    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c2816f-5dfxh" event={"ID":"824e8c00-991b-43a9-a368-04b7baa1e746","Type":"ContainerDied","Data":"66145bb3ea18ba9db4a1dc1730ffb3740b59ec1d2a596b39442c6a9d9d2d3662"}
Nov 03 01:00:03.483759 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:03.483716    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk" podStartSLOduration=3.483704901 podStartE2EDuration="3.483704901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:03.481026182 +0000 UTC m=+3286.360610226" watchObservedRunningTime="2026-11-03 01:00:03.483704901 +0000 UTC m=+3286.363288962"
Nov 03 01:00:03.520505 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:03.520459    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw" podStartSLOduration=3.520444473 podStartE2EDuration="3.520444473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:03.518060347 +0000 UTC m=+3286.397644389" watchObservedRunningTime="2026-11-03 01:00:03.520444473 +0000 UTC m=+3286.400028516"
Nov 03 01:00:03.552277 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:03.552230    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv" podStartSLOduration=3.552214658 podStartE2EDuration="3.552214658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:03.551848688 +0000 UTC m=+3286.431432732" watchObservedRunningTime="2026-11-03 01:00:03.552214658 +0000 UTC m=+3286.431798701"
Nov 03 01:00:03.626067 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:03.626019    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2" podStartSLOduration=3.626005438 podStartE2EDuration="3.626005438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:03.625610346 +0000 UTC m=+3286.505194403" watchObservedRunningTime="2026-11-03 01:00:03.626005438 +0000 UTC m=+3286.505589481"
Nov 03 01:00:04.469058 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.469017    5012 generic.go:356] "Generic (PLEG): container finished" podID="7eea71f4-f5d2-4530-9105-8e4e4c77c4fd" containerID="eb0b6ff3f2bd530c261b99782a2099c25337249721ac2771b885e3921431fe57" exitCode=60
Nov 03 01:00:04.469645 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.469096    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2" event={"ID":"7eea71f4-f5d2-4530-9105-8e4e4c77c4fd","Type":"ContainerDied","Data":"eb0b6ff3f2bd530c261b99782a2099c25337249721ac2771b885e3921431fe57"}
Nov 03 01:00:04.469645 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.469165    5012 scope.go:122] "RemoveContainer" containerID="d1fd39927190fe05a1e156ca940bf92943d45ff5ad87559b6cce545aed7fc0db"
Nov 03 01:00:04.469645 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.469442    5012 scope.go:122] "RemoveContainer" containerID="eb0b6ff3f2bd530c261b99782a2099c25337249721ac2771b885e3921431fe57"
Nov 03 01:00:04.469943 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:00:04.469895    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2_odh-ai-gateway-infra(7eea71f4-f5d2-4530-9105-8e4e4c77c4fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2" podUID="7eea71f4-f5d2-4530-9105-8e4e4c77c4fd"
Nov 03 01:00:04.471556 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.471532    5012 generic.go:356] "Generic (PLEG): container finished" podID="6b934863-00e3-4318-9d21-8bb7b64d9761" containerID="d0fecac02804c6d4699a7f60115a8fab5dbc3a78718034b8b323b6354531f970" exitCode=60
Nov 03 01:00:04.471695 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.471602    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw" event={"ID":"6b934863-00e3-4318-9d21-8bb7b64d9761","Type":"ContainerDied","Data":"d0fecac02804c6d4699a7f60115a8fab5dbc3a78718034b8b323b6354531f970"}
Nov 03 01:00:04.471985 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.471965    5012 scope.go:122] "RemoveContainer" containerID="d0fecac02804c6d4699a7f60115a8fab5dbc3a78718034b8b323b6354531f970"
Nov 03 01:00:04.472367 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:00:04.472344    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw_odh-ai-gateway-infra(6b934863-00e3-4318-9d21-8bb7b64d9761)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw" podUID="6b934863-00e3-4318-9d21-8bb7b64d9761"
Nov 03 01:00:04.474102 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.474074    5012 generic.go:356] "Generic (PLEG): container finished" podID="703067f2-bbcf-475f-8ed2-7146f5ad5f6b" containerID="5c3cc77e87c6639fd89a04e576374c18d3a7e12e69a494bd5ee390b012a3d66c" exitCode=60
Nov 03 01:00:04.474245 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.474191    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv" event={"ID":"703067f2-bbcf-475f-8ed2-7146f5ad5f6b","Type":"ContainerDied","Data":"5c3cc77e87c6639fd89a04e576374c18d3a7e12e69a494bd5ee390b012a3d66c"}
Nov 03 01:00:04.475122 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.474652    5012 scope.go:122] "RemoveContainer" containerID="5c3cc77e87c6639fd89a04e576374c18d3a7e12e69a494bd5ee390b012a3d66c"
Nov 03 01:00:04.475122 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:00:04.475075    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv_odh-ai-gateway-infra(703067f2-bbcf-475f-8ed2-7146f5ad5f6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv" podUID="703067f2-bbcf-475f-8ed2-7146f5ad5f6b"
Nov 03 01:00:04.476754 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.476728    5012 generic.go:356] "Generic (PLEG): container finished" podID="f1faca4e-fef0-4781-a61b-52d107115094" containerID="ac9913e205a5f6d62de91beeede1f3f890952b4c6967d20988aeef0727cca8fc" exitCode=60
Nov 03 01:00:04.476875 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.476806    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk" event={"ID":"f1faca4e-fef0-4781-a61b-52d107115094","Type":"ContainerDied","Data":"ac9913e205a5f6d62de91beeede1f3f890952b4c6967d20988aeef0727cca8fc"}
Nov 03 01:00:04.477082 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.477063    5012 scope.go:122] "RemoveContainer" containerID="ac9913e205a5f6d62de91beeede1f3f890952b4c6967d20988aeef0727cca8fc"
Nov 03 01:00:04.477446 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:00:04.477420    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk_odh-ai-gateway-infra(f1faca4e-fef0-4781-a61b-52d107115094)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk" podUID="f1faca4e-fef0-4781-a61b-52d107115094"
Nov 03 01:00:04.485877 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.485856    5012 scope.go:122] "RemoveContainer" containerID="e10cc87fd1cb077913501f84244ef3918030e8d621ca922d7e3917d934764145"
Nov 03 01:00:04.505624 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.505552    5012 scope.go:122] "RemoveContainer" containerID="4b4aadca9f12f2bee6da4f73aefffa084e49f022a31d9e899b65cf53b80d66a9"
Nov 03 01:00:04.525816 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.525782    5012 scope.go:122] "RemoveContainer" containerID="49c9038bc4cc7ba122ceb093e20bf7f6b00508d66241491f84d4d85e2613fae9"
Nov 03 01:00:04.706095 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.706068    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c2816f-5dfxh"
Nov 03 01:00:04.816819 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.816723    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/824e8c00-991b-43a9-a368-04b7baa1e746-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824e8c00-991b-43a9-a368-04b7baa1e746-maas-api-openshift-ca\") pod \"824e8c00-991b-43a9-a368-04b7baa1e746\" (UID: \"824e8c00-991b-43a9-a368-04b7baa1e746\") "
Nov 03 01:00:04.817042 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.816882    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824e8c00-991b-43a9-a368-04b7baa1e746-kube-api-access-k7965\" (UniqueName: \"kubernetes.io/projected/824e8c00-991b-43a9-a368-04b7baa1e746-kube-api-access-k7965\") pod \"824e8c00-991b-43a9-a368-04b7baa1e746\" (UID: \"824e8c00-991b-43a9-a368-04b7baa1e746\") "
Nov 03 01:00:04.817203 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.817158    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/824e8c00-991b-43a9-a368-04b7baa1e746-maas-api-openshift-ca" pod "824e8c00-991b-43a9-a368-04b7baa1e746" (UID: "824e8c00-991b-43a9-a368-04b7baa1e746"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:04.817288 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.817223    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/824e8c00-991b-43a9-a368-04b7baa1e746-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824e8c00-991b-43a9-a368-04b7baa1e746-maas-api-xks-ca\") pod \"824e8c00-991b-43a9-a368-04b7baa1e746\" (UID: \"824e8c00-991b-43a9-a368-04b7baa1e746\") "
Nov 03 01:00:04.818012 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.817981    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824e8c00-991b-43a9-a368-04b7baa1e746-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:04.819751 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.819720    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/824e8c00-991b-43a9-a368-04b7baa1e746-maas-api-xks-ca" pod "824e8c00-991b-43a9-a368-04b7baa1e746" (UID: "824e8c00-991b-43a9-a368-04b7baa1e746"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:00:04.819861 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.819792    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824e8c00-991b-43a9-a368-04b7baa1e746-kube-api-access-k7965" pod "824e8c00-991b-43a9-a368-04b7baa1e746" (UID: "824e8c00-991b-43a9-a368-04b7baa1e746"). InnerVolumeSpecName "kube-api-access-k7965". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:04.920159 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.920116    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824e8c00-991b-43a9-a368-04b7baa1e746-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:04.920159 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:04.920154    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7965\" (UniqueName: \"kubernetes.io/projected/824e8c00-991b-43a9-a368-04b7baa1e746-kube-api-access-k7965\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:05.490287 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:05.490245    5012 scope.go:122] "RemoveContainer" containerID="eb0b6ff3f2bd530c261b99782a2099c25337249721ac2771b885e3921431fe57"
Nov 03 01:00:05.490783 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:00:05.490707    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2_odh-ai-gateway-infra(7eea71f4-f5d2-4530-9105-8e4e4c77c4fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2" podUID="7eea71f4-f5d2-4530-9105-8e4e4c77c4fd"
Nov 03 01:00:05.492120 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:05.492093    5012 scope.go:122] "RemoveContainer" containerID="d0fecac02804c6d4699a7f60115a8fab5dbc3a78718034b8b323b6354531f970"
Nov 03 01:00:05.492861 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:00:05.492487    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw_odh-ai-gateway-infra(6b934863-00e3-4318-9d21-8bb7b64d9761)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw" podUID="6b934863-00e3-4318-9d21-8bb7b64d9761"
Nov 03 01:00:05.493988 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:05.493967    5012 scope.go:122] "RemoveContainer" containerID="5c3cc77e87c6639fd89a04e576374c18d3a7e12e69a494bd5ee390b012a3d66c"
Nov 03 01:00:05.494328 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:00:05.494307    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv_odh-ai-gateway-infra(703067f2-bbcf-475f-8ed2-7146f5ad5f6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv" podUID="703067f2-bbcf-475f-8ed2-7146f5ad5f6b"
Nov 03 01:00:05.495303 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:05.495286    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c2816f-5dfxh"
Nov 03 01:00:05.495390 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:05.495292    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c2816f-5dfxh" event={"ID":"824e8c00-991b-43a9-a368-04b7baa1e746","Type":"ContainerDied","Data":"d7484ca51fdcdd4e4ba1c6052669b98ceaea73a73f233f6ed4fc90610c863afb"}
Nov 03 01:00:05.495390 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:05.495339    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7484ca51fdcdd4e4ba1c6052669b98ceaea73a73f233f6ed4fc90610c863afb"
Nov 03 01:00:05.497803 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:05.497785    5012 scope.go:122] "RemoveContainer" containerID="ac9913e205a5f6d62de91beeede1f3f890952b4c6967d20988aeef0727cca8fc"
Nov 03 01:00:05.498030 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:00:05.498010    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk_odh-ai-gateway-infra(f1faca4e-fef0-4781-a61b-52d107115094)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk" podUID="f1faca4e-fef0-4781-a61b-52d107115094"
Nov 03 01:00:07.365688 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:00:07.365652    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:00:15.236411 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:15.236377    5012 scope.go:122] "RemoveContainer" containerID="d0fecac02804c6d4699a7f60115a8fab5dbc3a78718034b8b323b6354531f970"
Nov 03 01:00:15.236901 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:15.236557    5012 scope.go:122] "RemoveContainer" containerID="5c3cc77e87c6639fd89a04e576374c18d3a7e12e69a494bd5ee390b012a3d66c"
Nov 03 01:00:16.237702 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:16.237668    5012 scope.go:122] "RemoveContainer" containerID="eb0b6ff3f2bd530c261b99782a2099c25337249721ac2771b885e3921431fe57"
Nov 03 01:00:16.599345 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:16.599314    5012 generic.go:356] "Generic (PLEG): container finished" podID="6b934863-00e3-4318-9d21-8bb7b64d9761" containerID="73563c2dd71f08543b9d708d9a805ee01d23a395ddd2817d3bb7f24754263f6e" exitCode=60
Nov 03 01:00:16.599479 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:16.599396    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw" event={"ID":"6b934863-00e3-4318-9d21-8bb7b64d9761","Type":"ContainerDied","Data":"73563c2dd71f08543b9d708d9a805ee01d23a395ddd2817d3bb7f24754263f6e"}
Nov 03 01:00:16.599479 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:16.599463    5012 scope.go:122] "RemoveContainer" containerID="d0fecac02804c6d4699a7f60115a8fab5dbc3a78718034b8b323b6354531f970"
Nov 03 01:00:16.599873 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:16.599853    5012 scope.go:122] "RemoveContainer" containerID="73563c2dd71f08543b9d708d9a805ee01d23a395ddd2817d3bb7f24754263f6e"
Nov 03 01:00:16.600268 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:00:16.600238    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw_odh-ai-gateway-infra(6b934863-00e3-4318-9d21-8bb7b64d9761)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw" podUID="6b934863-00e3-4318-9d21-8bb7b64d9761"
Nov 03 01:00:16.601868 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:16.601845    5012 generic.go:356] "Generic (PLEG): container finished" podID="703067f2-bbcf-475f-8ed2-7146f5ad5f6b" containerID="f3b188e0de0f9f078adb8b64e04462b00ad75fd18d80b4707ee3f23ffafd19c0" exitCode=60
Nov 03 01:00:16.601986 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:16.601873    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv" event={"ID":"703067f2-bbcf-475f-8ed2-7146f5ad5f6b","Type":"ContainerDied","Data":"f3b188e0de0f9f078adb8b64e04462b00ad75fd18d80b4707ee3f23ffafd19c0"}
Nov 03 01:00:16.602288 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:16.602272    5012 scope.go:122] "RemoveContainer" containerID="f3b188e0de0f9f078adb8b64e04462b00ad75fd18d80b4707ee3f23ffafd19c0"
Nov 03 01:00:16.602578 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:00:16.602558    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv_odh-ai-gateway-infra(703067f2-bbcf-475f-8ed2-7146f5ad5f6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv" podUID="703067f2-bbcf-475f-8ed2-7146f5ad5f6b"
Nov 03 01:00:16.613579 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:16.613532    5012 scope.go:122] "RemoveContainer" containerID="5c3cc77e87c6639fd89a04e576374c18d3a7e12e69a494bd5ee390b012a3d66c"
Nov 03 01:00:17.612518 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:17.612485    5012 generic.go:356] "Generic (PLEG): container finished" podID="7eea71f4-f5d2-4530-9105-8e4e4c77c4fd" containerID="8b42c3f10bb48dca955dbae161d90ce49bbb584278378631b5895e5ff6f61980" exitCode=60
Nov 03 01:00:17.613003 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:17.612554    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2" event={"ID":"7eea71f4-f5d2-4530-9105-8e4e4c77c4fd","Type":"ContainerDied","Data":"8b42c3f10bb48dca955dbae161d90ce49bbb584278378631b5895e5ff6f61980"}
Nov 03 01:00:17.613003 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:17.612642    5012 scope.go:122] "RemoveContainer" containerID="eb0b6ff3f2bd530c261b99782a2099c25337249721ac2771b885e3921431fe57"
Nov 03 01:00:17.613144 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:17.613054    5012 scope.go:122] "RemoveContainer" containerID="8b42c3f10bb48dca955dbae161d90ce49bbb584278378631b5895e5ff6f61980"
Nov 03 01:00:17.613437 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:00:17.613408    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2_odh-ai-gateway-infra(7eea71f4-f5d2-4530-9105-8e4e4c77c4fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2" podUID="7eea71f4-f5d2-4530-9105-8e4e4c77c4fd"
Nov 03 01:00:17.652294 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:17.652270    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw"]
Nov 03 01:00:17.686077 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:17.686050    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv"]
Nov 03 01:00:17.881044 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:17.880964    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw"
Nov 03 01:00:17.955869 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:17.955834    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv"
Nov 03 01:00:17.958299 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:17.958273    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b934863-00e3-4318-9d21-8bb7b64d9761-kube-api-access-xgvn2\" (UniqueName: \"kubernetes.io/projected/6b934863-00e3-4318-9d21-8bb7b64d9761-kube-api-access-xgvn2\") pod \"6b934863-00e3-4318-9d21-8bb7b64d9761\" (UID: \"6b934863-00e3-4318-9d21-8bb7b64d9761\") "
Nov 03 01:00:17.958454 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:17.958437    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b934863-00e3-4318-9d21-8bb7b64d9761-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b934863-00e3-4318-9d21-8bb7b64d9761-maas-api-service-ca\") pod \"6b934863-00e3-4318-9d21-8bb7b64d9761\" (UID: \"6b934863-00e3-4318-9d21-8bb7b64d9761\") "
Nov 03 01:00:17.958536 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:17.958480    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/703067f2-bbcf-475f-8ed2-7146f5ad5f6b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/703067f2-bbcf-475f-8ed2-7146f5ad5f6b-maas-api-service-ca\") pod \"703067f2-bbcf-475f-8ed2-7146f5ad5f6b\" (UID: \"703067f2-bbcf-475f-8ed2-7146f5ad5f6b\") "
Nov 03 01:00:17.958536 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:17.958528    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/703067f2-bbcf-475f-8ed2-7146f5ad5f6b-kube-api-access-m8z2t\" (UniqueName: \"kubernetes.io/projected/703067f2-bbcf-475f-8ed2-7146f5ad5f6b-kube-api-access-m8z2t\") pod \"703067f2-bbcf-475f-8ed2-7146f5ad5f6b\" (UID: \"703067f2-bbcf-475f-8ed2-7146f5ad5f6b\") "
Nov 03 01:00:17.958886 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:17.958848    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b934863-00e3-4318-9d21-8bb7b64d9761-maas-api-service-ca" pod "6b934863-00e3-4318-9d21-8bb7b64d9761" (UID: "6b934863-00e3-4318-9d21-8bb7b64d9761"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:17.959040 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:17.958892    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/703067f2-bbcf-475f-8ed2-7146f5ad5f6b-maas-api-service-ca" pod "703067f2-bbcf-475f-8ed2-7146f5ad5f6b" (UID: "703067f2-bbcf-475f-8ed2-7146f5ad5f6b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:17.959264 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:17.959244    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b934863-00e3-4318-9d21-8bb7b64d9761-maas-api-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:17.959315 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:17.959271    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/703067f2-bbcf-475f-8ed2-7146f5ad5f6b-maas-api-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:17.960727 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:17.960699    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/703067f2-bbcf-475f-8ed2-7146f5ad5f6b-kube-api-access-m8z2t" pod "703067f2-bbcf-475f-8ed2-7146f5ad5f6b" (UID: "703067f2-bbcf-475f-8ed2-7146f5ad5f6b"). InnerVolumeSpecName "kube-api-access-m8z2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:17.961015 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:17.960995    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b934863-00e3-4318-9d21-8bb7b64d9761-kube-api-access-xgvn2" pod "6b934863-00e3-4318-9d21-8bb7b64d9761" (UID: "6b934863-00e3-4318-9d21-8bb7b64d9761"). InnerVolumeSpecName "kube-api-access-xgvn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:18.061344 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:18.061300    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgvn2\" (UniqueName: \"kubernetes.io/projected/6b934863-00e3-4318-9d21-8bb7b64d9761-kube-api-access-xgvn2\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:18.061344 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:18.061345    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8z2t\" (UniqueName: \"kubernetes.io/projected/703067f2-bbcf-475f-8ed2-7146f5ad5f6b-kube-api-access-m8z2t\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:18.236566 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:18.236483    5012 scope.go:122] "RemoveContainer" containerID="ac9913e205a5f6d62de91beeede1f3f890952b4c6967d20988aeef0727cca8fc"
Nov 03 01:00:18.439209 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:18.439168    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 01:00:18.442860 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:18.442831    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 01:00:18.443784 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:18.443765    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 01:00:18.447084 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:18.447061    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 01:00:18.628941 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:18.628884    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw"
Nov 03 01:00:18.628941 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:18.628887    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw" event={"ID":"6b934863-00e3-4318-9d21-8bb7b64d9761","Type":"ContainerDied","Data":"32c23bd0ae70fc2ee8ac90dfb6a98e18ee5508627f615534cb888031e4ecbc77"}
Nov 03 01:00:18.629519 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:18.628986    5012 scope.go:122] "RemoveContainer" containerID="73563c2dd71f08543b9d708d9a805ee01d23a395ddd2817d3bb7f24754263f6e"
Nov 03 01:00:18.630797 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:18.630768    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv" event={"ID":"703067f2-bbcf-475f-8ed2-7146f5ad5f6b","Type":"ContainerDied","Data":"4592c31e3ca138e8e0d58ffd6ccdebc03bb4ce429c26fe51088b8067bffe242c"}
Nov 03 01:00:18.630889 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:18.630863    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv"
Nov 03 01:00:18.643631 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:18.643613    5012 scope.go:122] "RemoveContainer" containerID="f3b188e0de0f9f078adb8b64e04462b00ad75fd18d80b4707ee3f23ffafd19c0"
Nov 03 01:00:18.672507 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:18.672484    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2"]
Nov 03 01:00:18.712376 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:18.712241    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv"]
Nov 03 01:00:18.721159 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:18.721135    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894460-cdvhv"]
Nov 03 01:00:18.768370 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:18.768304    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw"]
Nov 03 01:00:18.778437 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:18.776609    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894460-m25vw"]
Nov 03 01:00:18.868841 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:18.868807    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2"
Nov 03 01:00:18.873512 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:18.873486    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eea71f4-f5d2-4530-9105-8e4e4c77c4fd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eea71f4-f5d2-4530-9105-8e4e4c77c4fd-maas-api-service-ca\") pod \"7eea71f4-f5d2-4530-9105-8e4e4c77c4fd\" (UID: \"7eea71f4-f5d2-4530-9105-8e4e4c77c4fd\") "
Nov 03 01:00:18.873750 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:18.873733    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eea71f4-f5d2-4530-9105-8e4e4c77c4fd-kube-api-access-f2k76\" (UniqueName: \"kubernetes.io/projected/7eea71f4-f5d2-4530-9105-8e4e4c77c4fd-kube-api-access-f2k76\") pod \"7eea71f4-f5d2-4530-9105-8e4e4c77c4fd\" (UID: \"7eea71f4-f5d2-4530-9105-8e4e4c77c4fd\") "
Nov 03 01:00:18.873871 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:18.873849    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eea71f4-f5d2-4530-9105-8e4e4c77c4fd-maas-api-service-ca" pod "7eea71f4-f5d2-4530-9105-8e4e4c77c4fd" (UID: "7eea71f4-f5d2-4530-9105-8e4e4c77c4fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:18.874375 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:18.874354    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eea71f4-f5d2-4530-9105-8e4e4c77c4fd-maas-api-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:18.876008 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:18.875984    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eea71f4-f5d2-4530-9105-8e4e4c77c4fd-kube-api-access-f2k76" pod "7eea71f4-f5d2-4530-9105-8e4e4c77c4fd" (UID: "7eea71f4-f5d2-4530-9105-8e4e4c77c4fd"). InnerVolumeSpecName "kube-api-access-f2k76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:18.975953 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:18.975902    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2k76\" (UniqueName: \"kubernetes.io/projected/7eea71f4-f5d2-4530-9105-8e4e4c77c4fd-kube-api-access-f2k76\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:19.246619 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:19.246591    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b934863-00e3-4318-9d21-8bb7b64d9761" path="/var/lib/kubelet/pods/6b934863-00e3-4318-9d21-8bb7b64d9761/volumes"
Nov 03 01:00:19.247105 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:19.247090    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="703067f2-bbcf-475f-8ed2-7146f5ad5f6b" path="/var/lib/kubelet/pods/703067f2-bbcf-475f-8ed2-7146f5ad5f6b/volumes"
Nov 03 01:00:19.641724 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:19.641688    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2" event={"ID":"7eea71f4-f5d2-4530-9105-8e4e4c77c4fd","Type":"ContainerDied","Data":"d70cb75c0d8d69125de2d45c5a9f950d47c525b67d6d18a3112913f1a2f8e206"}
Nov 03 01:00:19.642243 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:19.641707    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2"
Nov 03 01:00:19.642243 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:19.641750    5012 scope.go:122] "RemoveContainer" containerID="8b42c3f10bb48dca955dbae161d90ce49bbb584278378631b5895e5ff6f61980"
Nov 03 01:00:19.646600 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:19.646577    5012 generic.go:356] "Generic (PLEG): container finished" podID="f1faca4e-fef0-4781-a61b-52d107115094" containerID="2a12bb7fbafdebb44236c49d2ef9d0cf83eb087295c88f6b85e30155b9ac61b6" exitCode=60
Nov 03 01:00:19.646748 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:19.646639    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk" event={"ID":"f1faca4e-fef0-4781-a61b-52d107115094","Type":"ContainerDied","Data":"2a12bb7fbafdebb44236c49d2ef9d0cf83eb087295c88f6b85e30155b9ac61b6"}
Nov 03 01:00:19.647091 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:19.647070    5012 scope.go:122] "RemoveContainer" containerID="2a12bb7fbafdebb44236c49d2ef9d0cf83eb087295c88f6b85e30155b9ac61b6"
Nov 03 01:00:19.647376 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:00:19.647357    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk_odh-ai-gateway-infra(f1faca4e-fef0-4781-a61b-52d107115094)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk" podUID="f1faca4e-fef0-4781-a61b-52d107115094"
Nov 03 01:00:19.653954 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:19.653938    5012 scope.go:122] "RemoveContainer" containerID="ac9913e205a5f6d62de91beeede1f3f890952b4c6967d20988aeef0727cca8fc"
Nov 03 01:00:19.690982 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:19.690956    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2"]
Nov 03 01:00:19.698028 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:19.698007    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894460-8f4k2"]
Nov 03 01:00:20.743110 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:20.743078    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk"]
Nov 03 01:00:20.938965 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:20.938940    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk"
Nov 03 01:00:21.001339 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:21.001262    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1faca4e-fef0-4781-a61b-52d107115094-kube-api-access-wbp2r\" (UniqueName: \"kubernetes.io/projected/f1faca4e-fef0-4781-a61b-52d107115094-kube-api-access-wbp2r\") pod \"f1faca4e-fef0-4781-a61b-52d107115094\" (UID: \"f1faca4e-fef0-4781-a61b-52d107115094\") "
Nov 03 01:00:21.001642 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:21.001614    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1faca4e-fef0-4781-a61b-52d107115094-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1faca4e-fef0-4781-a61b-52d107115094-maas-api-service-ca\") pod \"f1faca4e-fef0-4781-a61b-52d107115094\" (UID: \"f1faca4e-fef0-4781-a61b-52d107115094\") "
Nov 03 01:00:21.002077 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:21.002048    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1faca4e-fef0-4781-a61b-52d107115094-maas-api-service-ca" pod "f1faca4e-fef0-4781-a61b-52d107115094" (UID: "f1faca4e-fef0-4781-a61b-52d107115094"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:21.002286 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:21.002266    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1faca4e-fef0-4781-a61b-52d107115094-maas-api-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:21.004134 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:21.004099    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1faca4e-fef0-4781-a61b-52d107115094-kube-api-access-wbp2r" pod "f1faca4e-fef0-4781-a61b-52d107115094" (UID: "f1faca4e-fef0-4781-a61b-52d107115094"). InnerVolumeSpecName "kube-api-access-wbp2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:21.103259 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:21.103231    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbp2r\" (UniqueName: \"kubernetes.io/projected/f1faca4e-fef0-4781-a61b-52d107115094-kube-api-access-wbp2r\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:21.248459 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:21.248423    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eea71f4-f5d2-4530-9105-8e4e4c77c4fd" path="/var/lib/kubelet/pods/7eea71f4-f5d2-4530-9105-8e4e4c77c4fd/volumes"
Nov 03 01:00:21.668810 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:21.668782    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk"
Nov 03 01:00:21.668810 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:21.668797    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk" event={"ID":"f1faca4e-fef0-4781-a61b-52d107115094","Type":"ContainerDied","Data":"1cc83be1c67e6c78e47e957065b6963ac16269a7da60d84517f325636f3020c2"}
Nov 03 01:00:21.669054 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:21.668853    5012 scope.go:122] "RemoveContainer" containerID="2a12bb7fbafdebb44236c49d2ef9d0cf83eb087295c88f6b85e30155b9ac61b6"
Nov 03 01:00:21.717540 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:21.717511    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk"]
Nov 03 01:00:21.730716 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:21.730688    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894460-ddbkk"]
Nov 03 01:00:22.416415 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:22.416376    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"]
Nov 03 01:00:23.246992 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:23.246957    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1faca4e-fef0-4781-a61b-52d107115094" path="/var/lib/kubelet/pods/f1faca4e-fef0-4781-a61b-52d107115094/volumes"
Nov 03 01:00:37.404973 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:00:37.404924    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:00:37.438973 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.438932    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8" podUID="31151eab-95c9-4158-b86b-5708cf5c7329" containerName="main" containerID="cri-o://b5599386603a814ebb5d579e9c002a3b17ad3f947018c387c563139e356a4d8f" gracePeriod=45
Nov 03 01:00:37.489153 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.489113    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8" podUID="31151eab-95c9-4158-b86b-5708cf5c7329" containerName="main" probeResult="failure" output="Get \"https://10.132.1.15:8000/ready\": dial tcp 10.132.1.15:8000: connect: connection refused"
Nov 03 01:00:37.745804 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.745777    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 01:00:37.787665 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.787635    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-home\" (UniqueName: \"kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-home\") pod \"31151eab-95c9-4158-b86b-5708cf5c7329\" (UID: \"31151eab-95c9-4158-b86b-5708cf5c7329\") "
Nov 03 01:00:37.787797 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.787724    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31151eab-95c9-4158-b86b-5708cf5c7329-kube-api-access-bvzrj\" (UniqueName: \"kubernetes.io/projected/31151eab-95c9-4158-b86b-5708cf5c7329-kube-api-access-bvzrj\") pod \"31151eab-95c9-4158-b86b-5708cf5c7329\" (UID: \"31151eab-95c9-4158-b86b-5708cf5c7329\") "
Nov 03 01:00:37.787848 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.787810    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-model-cache\") pod \"31151eab-95c9-4158-b86b-5708cf5c7329\" (UID: \"31151eab-95c9-4158-b86b-5708cf5c7329\") "
Nov 03 01:00:37.787940 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.787845    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-dshm\" (UniqueName: \"kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-dshm\") pod \"31151eab-95c9-4158-b86b-5708cf5c7329\" (UID: \"31151eab-95c9-4158-b86b-5708cf5c7329\") "
Nov 03 01:00:37.787940 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.787884    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-tmp-dir\") pod \"31151eab-95c9-4158-b86b-5708cf5c7329\" (UID: \"31151eab-95c9-4158-b86b-5708cf5c7329\") "
Nov 03 01:00:37.787940 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.787901    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-home" pod "31151eab-95c9-4158-b86b-5708cf5c7329" (UID: "31151eab-95c9-4158-b86b-5708cf5c7329"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:00:37.788134 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.788051    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31151eab-95c9-4158-b86b-5708cf5c7329-tls-certs\" (UniqueName: \"kubernetes.io/secret/31151eab-95c9-4158-b86b-5708cf5c7329-tls-certs\") pod \"31151eab-95c9-4158-b86b-5708cf5c7329\" (UID: \"31151eab-95c9-4158-b86b-5708cf5c7329\") "
Nov 03 01:00:37.788196 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.788143    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-model-cache" pod "31151eab-95c9-4158-b86b-5708cf5c7329" (UID: "31151eab-95c9-4158-b86b-5708cf5c7329"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:00:37.788266 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.788217    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-tmp-dir" pod "31151eab-95c9-4158-b86b-5708cf5c7329" (UID: "31151eab-95c9-4158-b86b-5708cf5c7329"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:00:37.788665 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.788638    5012 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-model-cache\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:37.788665 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.788667    5012 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-tmp-dir\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:37.788834 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.788678    5012 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-home\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:37.790411 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.790388    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-dshm" pod "31151eab-95c9-4158-b86b-5708cf5c7329" (UID: "31151eab-95c9-4158-b86b-5708cf5c7329"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:00:37.790520 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.790451    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31151eab-95c9-4158-b86b-5708cf5c7329-kube-api-access-bvzrj" pod "31151eab-95c9-4158-b86b-5708cf5c7329" (UID: "31151eab-95c9-4158-b86b-5708cf5c7329"). InnerVolumeSpecName "kube-api-access-bvzrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:37.793539 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.793516    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31151eab-95c9-4158-b86b-5708cf5c7329-tls-certs" pod "31151eab-95c9-4158-b86b-5708cf5c7329" (UID: "31151eab-95c9-4158-b86b-5708cf5c7329"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:00:37.825571 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.825547    5012 generic.go:356] "Generic (PLEG): container finished" podID="31151eab-95c9-4158-b86b-5708cf5c7329" containerID="b5599386603a814ebb5d579e9c002a3b17ad3f947018c387c563139e356a4d8f" exitCode=0
Nov 03 01:00:37.825701 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.825603    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8" event={"ID":"31151eab-95c9-4158-b86b-5708cf5c7329","Type":"ContainerDied","Data":"b5599386603a814ebb5d579e9c002a3b17ad3f947018c387c563139e356a4d8f"}
Nov 03 01:00:37.825701 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.825631    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8" event={"ID":"31151eab-95c9-4158-b86b-5708cf5c7329","Type":"ContainerDied","Data":"c2db4563d168d24f18b5d82cb07a81b9195ceb7062168181a21c2a197f4e47c1"}
Nov 03 01:00:37.825701 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.825649    5012 scope.go:122] "RemoveContainer" containerID="b5599386603a814ebb5d579e9c002a3b17ad3f947018c387c563139e356a4d8f"
Nov 03 01:00:37.825701 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.825605    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"
Nov 03 01:00:37.841707 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.841667    5012 scope.go:122] "RemoveContainer" containerID="b5599386603a814ebb5d579e9c002a3b17ad3f947018c387c563139e356a4d8f"
Nov 03 01:00:37.842062 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:00:37.842036    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5599386603a814ebb5d579e9c002a3b17ad3f947018c387c563139e356a4d8f\": container with ID starting with b5599386603a814ebb5d579e9c002a3b17ad3f947018c387c563139e356a4d8f not found: ID does not exist" containerID="b5599386603a814ebb5d579e9c002a3b17ad3f947018c387c563139e356a4d8f"
Nov 03 01:00:37.842154 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.842073    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5599386603a814ebb5d579e9c002a3b17ad3f947018c387c563139e356a4d8f"} err="failed to get container status \"b5599386603a814ebb5d579e9c002a3b17ad3f947018c387c563139e356a4d8f\": rpc error: code = NotFound desc = could not find container \"b5599386603a814ebb5d579e9c002a3b17ad3f947018c387c563139e356a4d8f\": container with ID starting with b5599386603a814ebb5d579e9c002a3b17ad3f947018c387c563139e356a4d8f not found: ID does not exist"
Nov 03 01:00:37.874592 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.874571    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"]
Nov 03 01:00:37.880151 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.880131    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-726205/test-model-726205-kserve-7fb55cb5ff-trlf8"]
Nov 03 01:00:37.889753 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.889733    5012 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31151eab-95c9-4158-b86b-5708cf5c7329-dshm\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:37.889855 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.889757    5012 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31151eab-95c9-4158-b86b-5708cf5c7329-tls-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:37.889855 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:37.889771    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvzrj\" (UniqueName: \"kubernetes.io/projected/31151eab-95c9-4158-b86b-5708cf5c7329-kube-api-access-bvzrj\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:39.246402 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:39.246369    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31151eab-95c9-4158-b86b-5708cf5c7329" path="/var/lib/kubelet/pods/31151eab-95c9-4158-b86b-5708cf5c7329/volumes"
Nov 03 01:00:40.343227 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:40.343198    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"]
Nov 03 01:00:40.343634 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:40.343377    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp" podUID="48052f77-fb0d-4da5-9a5b-1f24988f1bad" containerName="istio-proxy" containerID="cri-o://3859ff9d9319b35280135850e6f13bc41b75717bcd58343bf7aefc6883619bc8" gracePeriod=30
Nov 03 01:00:45.649042 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.649008    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 01:00:45.740930 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.740887    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-credential-socket\") pod \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") "
Nov 03 01:00:45.741173 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.741021    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48052f77-fb0d-4da5-9a5b-1f24988f1bad-kube-api-access-crx7m\" (UniqueName: \"kubernetes.io/projected/48052f77-fb0d-4da5-9a5b-1f24988f1bad-kube-api-access-crx7m\") pod \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") "
Nov 03 01:00:45.741253 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.741184    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-podinfo\") pod \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") "
Nov 03 01:00:45.741253 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.741205    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-credential-socket" pod "48052f77-fb0d-4da5-9a5b-1f24988f1bad" (UID: "48052f77-fb0d-4da5-9a5b-1f24988f1bad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:00:45.741379 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.741291    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-token\" (UniqueName: \"kubernetes.io/projected/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-token\") pod \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") "
Nov 03 01:00:45.741379 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.741340    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-workload-socket\") pod \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") "
Nov 03 01:00:45.741494 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.741423    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-data\") pod \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") "
Nov 03 01:00:45.741494 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.741469    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-envoy\") pod \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") "
Nov 03 01:00:45.741610 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.741552    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istiod-ca-cert\") pod \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") "
Nov 03 01:00:45.741610 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.741587    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-workload-certs\") pod \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\" (UID: \"48052f77-fb0d-4da5-9a5b-1f24988f1bad\") "
Nov 03 01:00:45.741785 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.741725    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-workload-socket" pod "48052f77-fb0d-4da5-9a5b-1f24988f1bad" (UID: "48052f77-fb0d-4da5-9a5b-1f24988f1bad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:00:45.742717 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.742506    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:45.742717 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.742532    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:45.742905 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.742786    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-data" pod "48052f77-fb0d-4da5-9a5b-1f24988f1bad" (UID: "48052f77-fb0d-4da5-9a5b-1f24988f1bad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:00:45.743180 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.743035    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-workload-certs" pod "48052f77-fb0d-4da5-9a5b-1f24988f1bad" (UID: "48052f77-fb0d-4da5-9a5b-1f24988f1bad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:00:45.743180 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.743145    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istiod-ca-cert" pod "48052f77-fb0d-4da5-9a5b-1f24988f1bad" (UID: "48052f77-fb0d-4da5-9a5b-1f24988f1bad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:45.744037 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.744004    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-podinfo" pod "48052f77-fb0d-4da5-9a5b-1f24988f1bad" (UID: "48052f77-fb0d-4da5-9a5b-1f24988f1bad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:00:45.748356 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.748285    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-envoy" pod "48052f77-fb0d-4da5-9a5b-1f24988f1bad" (UID: "48052f77-fb0d-4da5-9a5b-1f24988f1bad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:00:45.748444 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.748398    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-token" pod "48052f77-fb0d-4da5-9a5b-1f24988f1bad" (UID: "48052f77-fb0d-4da5-9a5b-1f24988f1bad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:45.748444 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.748433    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48052f77-fb0d-4da5-9a5b-1f24988f1bad-kube-api-access-crx7m" pod "48052f77-fb0d-4da5-9a5b-1f24988f1bad" (UID: "48052f77-fb0d-4da5-9a5b-1f24988f1bad"). InnerVolumeSpecName "kube-api-access-crx7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:45.879629 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.879458    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crx7m\" (UniqueName: \"kubernetes.io/projected/48052f77-fb0d-4da5-9a5b-1f24988f1bad-kube-api-access-crx7m\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:45.879629 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.879500    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:45.879629 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.879515    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:45.879629 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.879529    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:45.879629 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.879542    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:45.879629 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.879554    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48052f77-fb0d-4da5-9a5b-1f24988f1bad-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:45.879629 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.879565    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48052f77-fb0d-4da5-9a5b-1f24988f1bad-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:00:45.908990 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.908949    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp" event={"ID":"48052f77-fb0d-4da5-9a5b-1f24988f1bad","Type":"ContainerDied","Data":"3859ff9d9319b35280135850e6f13bc41b75717bcd58343bf7aefc6883619bc8"}
Nov 03 01:00:45.909177 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.909016    5012 scope.go:122] "RemoveContainer" containerID="3859ff9d9319b35280135850e6f13bc41b75717bcd58343bf7aefc6883619bc8"
Nov 03 01:00:45.909177 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.908899    5012 generic.go:356] "Generic (PLEG): container finished" podID="48052f77-fb0d-4da5-9a5b-1f24988f1bad" containerID="3859ff9d9319b35280135850e6f13bc41b75717bcd58343bf7aefc6883619bc8" exitCode=0
Nov 03 01:00:45.909320 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.909198    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp" event={"ID":"48052f77-fb0d-4da5-9a5b-1f24988f1bad","Type":"ContainerDied","Data":"3045f9eef5aaabd3917f02435a371725b71411989c54f556cf51ce1e977063f4"}
Nov 03 01:00:45.909320 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.908968    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"
Nov 03 01:00:45.924259 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.924233    5012 scope.go:122] "RemoveContainer" containerID="3859ff9d9319b35280135850e6f13bc41b75717bcd58343bf7aefc6883619bc8"
Nov 03 01:00:45.924582 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:00:45.924554    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3859ff9d9319b35280135850e6f13bc41b75717bcd58343bf7aefc6883619bc8\": container with ID starting with 3859ff9d9319b35280135850e6f13bc41b75717bcd58343bf7aefc6883619bc8 not found: ID does not exist" containerID="3859ff9d9319b35280135850e6f13bc41b75717bcd58343bf7aefc6883619bc8"
Nov 03 01:00:45.924681 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.924588    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3859ff9d9319b35280135850e6f13bc41b75717bcd58343bf7aefc6883619bc8"} err="failed to get container status \"3859ff9d9319b35280135850e6f13bc41b75717bcd58343bf7aefc6883619bc8\": rpc error: code = NotFound desc = could not find container \"3859ff9d9319b35280135850e6f13bc41b75717bcd58343bf7aefc6883619bc8\": container with ID starting with 3859ff9d9319b35280135850e6f13bc41b75717bcd58343bf7aefc6883619bc8 not found: ID does not exist"
Nov 03 01:00:45.969936 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.969288    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"]
Nov 03 01:00:45.975141 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:45.975110    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zhtvp"]
Nov 03 01:00:47.248348 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:47.248298    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48052f77-fb0d-4da5-9a5b-1f24988f1bad" path="/var/lib/kubelet/pods/48052f77-fb0d-4da5-9a5b-1f24988f1bad/volumes"
Nov 03 01:00:49.183633 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.183602    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr"]
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.184635    5012 cpu_manager.go:414] "containerMap: removing container" podUID="703067f2-bbcf-475f-8ed2-7146f5ad5f6b" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.184654    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="703067f2-bbcf-475f-8ed2-7146f5ad5f6b" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.184680    5012 cpu_manager.go:414] "containerMap: removing container" podUID="48052f77-fb0d-4da5-9a5b-1f24988f1bad" containerName="istio-proxy"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.184688    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48052f77-fb0d-4da5-9a5b-1f24988f1bad" containerName="istio-proxy"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.184707    5012 cpu_manager.go:414] "containerMap: removing container" podUID="6b934863-00e3-4318-9d21-8bb7b64d9761" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.184715    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b934863-00e3-4318-9d21-8bb7b64d9761" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.184725    5012 cpu_manager.go:414] "containerMap: removing container" podUID="7eea71f4-f5d2-4530-9105-8e4e4c77c4fd" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.184732    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eea71f4-f5d2-4530-9105-8e4e4c77c4fd" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.184746    5012 cpu_manager.go:414] "containerMap: removing container" podUID="7eea71f4-f5d2-4530-9105-8e4e4c77c4fd" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.184753    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eea71f4-f5d2-4530-9105-8e4e4c77c4fd" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.184771    5012 cpu_manager.go:414] "containerMap: removing container" podUID="f1faca4e-fef0-4781-a61b-52d107115094" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.184777    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1faca4e-fef0-4781-a61b-52d107115094" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.184788    5012 cpu_manager.go:414] "containerMap: removing container" podUID="f1faca4e-fef0-4781-a61b-52d107115094" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.184795    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1faca4e-fef0-4781-a61b-52d107115094" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.184824    5012 cpu_manager.go:414] "containerMap: removing container" podUID="6b934863-00e3-4318-9d21-8bb7b64d9761" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.184831    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b934863-00e3-4318-9d21-8bb7b64d9761" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.184849    5012 cpu_manager.go:414] "containerMap: removing container" podUID="7eea71f4-f5d2-4530-9105-8e4e4c77c4fd" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.184856    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eea71f4-f5d2-4530-9105-8e4e4c77c4fd" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.184867    5012 cpu_manager.go:414] "containerMap: removing container" podUID="703067f2-bbcf-475f-8ed2-7146f5ad5f6b" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.184874    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="703067f2-bbcf-475f-8ed2-7146f5ad5f6b" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.184902    5012 cpu_manager.go:414] "containerMap: removing container" podUID="824e8c00-991b-43a9-a368-04b7baa1e746" containerName="revoke-keys"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.184925    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824e8c00-991b-43a9-a368-04b7baa1e746" containerName="revoke-keys"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.184946    5012 cpu_manager.go:414] "containerMap: removing container" podUID="31151eab-95c9-4158-b86b-5708cf5c7329" containerName="main"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.184953    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31151eab-95c9-4158-b86b-5708cf5c7329" containerName="main"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.185147    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="824e8c00-991b-43a9-a368-04b7baa1e746" containerName="revoke-keys"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.185165    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1faca4e-fef0-4781-a61b-52d107115094" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.185173    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="31151eab-95c9-4158-b86b-5708cf5c7329" containerName="main"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.185178    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="703067f2-bbcf-475f-8ed2-7146f5ad5f6b" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.185184    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b934863-00e3-4318-9d21-8bb7b64d9761" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.185192    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="703067f2-bbcf-475f-8ed2-7146f5ad5f6b" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.185201    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="48052f77-fb0d-4da5-9a5b-1f24988f1bad" containerName="istio-proxy"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.185207    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eea71f4-f5d2-4530-9105-8e4e4c77c4fd" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.185213    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eea71f4-f5d2-4530-9105-8e4e4c77c4fd" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.185220    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1faca4e-fef0-4781-a61b-52d107115094" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.185226    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1faca4e-fef0-4781-a61b-52d107115094" containerName="cleanup"
Nov 03 01:00:49.186087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.185239    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b934863-00e3-4318-9d21-8bb7b64d9761" containerName="cleanup"
Nov 03 01:00:49.189094 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.189078    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr"
Nov 03 01:00:49.224939 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.224889    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-70c7d6-serving-cert\""
Nov 03 01:00:49.230754 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.230731    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr"]
Nov 03 01:00:49.272415 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.272383    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ba67790-ca73-4724-b68f-21b645e1d4b3-maas-api-tls\") pod \"maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr\" (UID: \"8ba67790-ca73-4724-b68f-21b645e1d4b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr"
Nov 03 01:00:49.272578 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.272461    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba67790-ca73-4724-b68f-21b645e1d4b3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr\" (UID: \"8ba67790-ca73-4724-b68f-21b645e1d4b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr"
Nov 03 01:00:49.272795 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.272770    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5dnb\" (UniqueName: \"kubernetes.io/projected/8ba67790-ca73-4724-b68f-21b645e1d4b3-kube-api-access-z5dnb\") pod \"maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr\" (UID: \"8ba67790-ca73-4724-b68f-21b645e1d4b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr"
Nov 03 01:00:49.273091 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.273060    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ba67790-ca73-4724-b68f-21b645e1d4b3-metrics-certs\") pod \"maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr\" (UID: \"8ba67790-ca73-4724-b68f-21b645e1d4b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr"
Nov 03 01:00:49.374448 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.374417    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ba67790-ca73-4724-b68f-21b645e1d4b3-maas-api-tls\") pod \"maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr\" (UID: \"8ba67790-ca73-4724-b68f-21b645e1d4b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr"
Nov 03 01:00:49.374622 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.374478    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba67790-ca73-4724-b68f-21b645e1d4b3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr\" (UID: \"8ba67790-ca73-4724-b68f-21b645e1d4b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr"
Nov 03 01:00:49.374622 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.374558    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5dnb\" (UniqueName: \"kubernetes.io/projected/8ba67790-ca73-4724-b68f-21b645e1d4b3-kube-api-access-z5dnb\") pod \"maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr\" (UID: \"8ba67790-ca73-4724-b68f-21b645e1d4b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr"
Nov 03 01:00:49.374781 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.374763    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ba67790-ca73-4724-b68f-21b645e1d4b3-metrics-certs\") pod \"maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr\" (UID: \"8ba67790-ca73-4724-b68f-21b645e1d4b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr"
Nov 03 01:00:49.375298 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.375268    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba67790-ca73-4724-b68f-21b645e1d4b3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr\" (UID: \"8ba67790-ca73-4724-b68f-21b645e1d4b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr"
Nov 03 01:00:49.377192 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.377162    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ba67790-ca73-4724-b68f-21b645e1d4b3-maas-api-tls\") pod \"maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr\" (UID: \"8ba67790-ca73-4724-b68f-21b645e1d4b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr"
Nov 03 01:00:49.377329 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.377287    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ba67790-ca73-4724-b68f-21b645e1d4b3-metrics-certs\") pod \"maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr\" (UID: \"8ba67790-ca73-4724-b68f-21b645e1d4b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr"
Nov 03 01:00:49.392457 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.392433    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5dnb\" (UniqueName: \"kubernetes.io/projected/8ba67790-ca73-4724-b68f-21b645e1d4b3-kube-api-access-z5dnb\") pod \"maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr\" (UID: \"8ba67790-ca73-4724-b68f-21b645e1d4b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr"
Nov 03 01:00:49.504477 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.504261    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr"
Nov 03 01:00:49.826759 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:00:49.826727    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba67790_ca73_4724_b68f_21b645e1d4b3.slice/crio-a71280b575fee0110ae970e398dd00b590521a813e7c45fbceeb27a78fa61631 WatchSource:0}: Error finding container a71280b575fee0110ae970e398dd00b590521a813e7c45fbceeb27a78fa61631: Status 404 returned error can't find the container with id a71280b575fee0110ae970e398dd00b590521a813e7c45fbceeb27a78fa61631
Nov 03 01:00:49.827296 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.827220    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr"]
Nov 03 01:00:49.828172 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.828154    5012 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 01:00:49.983061 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:49.983019    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr" event={"ID":"8ba67790-ca73-4724-b68f-21b645e1d4b3","Type":"ContainerStarted","Data":"a71280b575fee0110ae970e398dd00b590521a813e7c45fbceeb27a78fa61631"}
Nov 03 01:00:50.993813 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:50.993770    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr" event={"ID":"8ba67790-ca73-4724-b68f-21b645e1d4b3","Type":"ContainerStarted","Data":"d3ebc5fa4a0afc69fee7f0271cfba9e0fb1c1bef316130b1a4dee84b0c89da45"}
Nov 03 01:00:50.994313 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:50.993854    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr"
Nov 03 01:00:51.034304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:51.034253    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr" podStartSLOduration=1.6090674310000002 podStartE2EDuration="2.034237744s" totalImagesPullingTime="425.170313ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:51.031326987 +0000 UTC m=+3333.910911041" watchObservedRunningTime="2026-11-03 01:00:51.034237744 +0000 UTC m=+3333.913821791"
Nov 03 01:00:55.557812 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:55.557776    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"]
Nov 03 01:00:55.558315 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:55.558006    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r" podUID="8e3746c4-7a9a-470e-a4f2-4b48f8863080" containerName="istio-proxy" containerID="cri-o://8435a451d41acfbc207c83134b15f7152fc83aff82e844a30ae048a2abb1d35b" gracePeriod=30
Nov 03 01:00:57.013638 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:00:57.013606    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr"
Nov 03 01:01:00.849451 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.849424    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 01:01:00.890638 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.890613    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-envoy\") pod \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") "
Nov 03 01:01:00.890764 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.890652    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-token\" (UniqueName: \"kubernetes.io/projected/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-token\") pod \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") "
Nov 03 01:01:00.890807 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.890785    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-data\") pod \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") "
Nov 03 01:01:00.890855 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.890822    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-workload-certs\") pod \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") "
Nov 03 01:01:00.890983 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.890965    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-workload-socket\") pod \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") "
Nov 03 01:01:00.891080 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.891038    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istiod-ca-cert\") pod \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") "
Nov 03 01:01:00.891134 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.891089    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-data" pod "8e3746c4-7a9a-470e-a4f2-4b48f8863080" (UID: "8e3746c4-7a9a-470e-a4f2-4b48f8863080"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:01:00.891134 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.891122    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-workload-certs" pod "8e3746c4-7a9a-470e-a4f2-4b48f8863080" (UID: "8e3746c4-7a9a-470e-a4f2-4b48f8863080"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:01:00.891244 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.891165    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-credential-socket\") pod \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") "
Nov 03 01:01:00.891244 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.891207    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e3746c4-7a9a-470e-a4f2-4b48f8863080-kube-api-access-svglc\" (UniqueName: \"kubernetes.io/projected/8e3746c4-7a9a-470e-a4f2-4b48f8863080-kube-api-access-svglc\") pod \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") "
Nov 03 01:01:00.891244 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.891239    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-podinfo\") pod \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\" (UID: \"8e3746c4-7a9a-470e-a4f2-4b48f8863080\") "
Nov 03 01:01:00.891460 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.891238    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-workload-socket" pod "8e3746c4-7a9a-470e-a4f2-4b48f8863080" (UID: "8e3746c4-7a9a-470e-a4f2-4b48f8863080"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:01:00.891460 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.891399    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-credential-socket" pod "8e3746c4-7a9a-470e-a4f2-4b48f8863080" (UID: "8e3746c4-7a9a-470e-a4f2-4b48f8863080"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:01:00.891578 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.891494    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istiod-ca-cert" pod "8e3746c4-7a9a-470e-a4f2-4b48f8863080" (UID: "8e3746c4-7a9a-470e-a4f2-4b48f8863080"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:01:00.891868 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.891842    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:01:00.891988 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.891873    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:01:00.891988 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.891887    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:01:00.891988 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.891900    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:01:00.891988 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.891941    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:01:00.893082 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.893057    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-envoy" pod "8e3746c4-7a9a-470e-a4f2-4b48f8863080" (UID: "8e3746c4-7a9a-470e-a4f2-4b48f8863080"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:01:00.893329 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.893310    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-token" pod "8e3746c4-7a9a-470e-a4f2-4b48f8863080" (UID: "8e3746c4-7a9a-470e-a4f2-4b48f8863080"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:01:00.893389 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.893345    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3746c4-7a9a-470e-a4f2-4b48f8863080-kube-api-access-svglc" pod "8e3746c4-7a9a-470e-a4f2-4b48f8863080" (UID: "8e3746c4-7a9a-470e-a4f2-4b48f8863080"). InnerVolumeSpecName "kube-api-access-svglc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:01:00.893535 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.893503    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-podinfo" pod "8e3746c4-7a9a-470e-a4f2-4b48f8863080" (UID: "8e3746c4-7a9a-470e-a4f2-4b48f8863080"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:01:00.993052 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.993021    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svglc\" (UniqueName: \"kubernetes.io/projected/8e3746c4-7a9a-470e-a4f2-4b48f8863080-kube-api-access-svglc\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:01:00.993052 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.993047    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:01:00.993052 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.993058    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:01:00.993270 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:00.993066    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3746c4-7a9a-470e-a4f2-4b48f8863080-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:01:01.140365 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:01.140333    5012 generic.go:356] "Generic (PLEG): container finished" podID="8e3746c4-7a9a-470e-a4f2-4b48f8863080" containerID="8435a451d41acfbc207c83134b15f7152fc83aff82e844a30ae048a2abb1d35b" exitCode=0
Nov 03 01:01:01.140534 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:01.140384    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"
Nov 03 01:01:01.140534 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:01.140417    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r" event={"ID":"8e3746c4-7a9a-470e-a4f2-4b48f8863080","Type":"ContainerDied","Data":"8435a451d41acfbc207c83134b15f7152fc83aff82e844a30ae048a2abb1d35b"}
Nov 03 01:01:01.140534 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:01.140475    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r" event={"ID":"8e3746c4-7a9a-470e-a4f2-4b48f8863080","Type":"ContainerDied","Data":"4a7e432fd85cab91591af94d581f8d6889b4cca07cbd8a0c6542f6690f786d46"}
Nov 03 01:01:01.140534 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:01.140504    5012 scope.go:122] "RemoveContainer" containerID="8435a451d41acfbc207c83134b15f7152fc83aff82e844a30ae048a2abb1d35b"
Nov 03 01:01:01.153689 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:01.153668    5012 scope.go:122] "RemoveContainer" containerID="8435a451d41acfbc207c83134b15f7152fc83aff82e844a30ae048a2abb1d35b"
Nov 03 01:01:01.153974 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:01:01.153955    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8435a451d41acfbc207c83134b15f7152fc83aff82e844a30ae048a2abb1d35b\": container with ID starting with 8435a451d41acfbc207c83134b15f7152fc83aff82e844a30ae048a2abb1d35b not found: ID does not exist" containerID="8435a451d41acfbc207c83134b15f7152fc83aff82e844a30ae048a2abb1d35b"
Nov 03 01:01:01.154038 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:01.153982    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8435a451d41acfbc207c83134b15f7152fc83aff82e844a30ae048a2abb1d35b"} err="failed to get container status \"8435a451d41acfbc207c83134b15f7152fc83aff82e844a30ae048a2abb1d35b\": rpc error: code = NotFound desc = could not find container \"8435a451d41acfbc207c83134b15f7152fc83aff82e844a30ae048a2abb1d35b\": container with ID starting with 8435a451d41acfbc207c83134b15f7152fc83aff82e844a30ae048a2abb1d35b not found: ID does not exist"
Nov 03 01:01:01.191326 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:01.191301    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"]
Nov 03 01:01:01.195996 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:01.195977    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-cp69r"]
Nov 03 01:01:01.247149 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:01.247117    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e3746c4-7a9a-470e-a4f2-4b48f8863080" path="/var/lib/kubelet/pods/8e3746c4-7a9a-470e-a4f2-4b48f8863080/volumes"
Nov 03 01:01:07.365929 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:01:07.365880    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:01:30.912083 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:30.911953    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"]
Nov 03 01:01:30.912592 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:30.912234    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2" podUID="8e95dc58-cf96-48fd-91b0-7ce6156ed573" containerName="istio-proxy" containerID="cri-o://61bc9e3fc1f70ee519b5f6fd6a231acaa578cc5e69dd2169ca66080cd07dd290" gracePeriod=30
Nov 03 01:01:36.196571 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.196509    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 01:01:36.331734 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.329147    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-podinfo\") pod \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") "
Nov 03 01:01:36.331734 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.329233    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-envoy\") pod \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") "
Nov 03 01:01:36.331734 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.329277    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-workload-socket\") pod \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") "
Nov 03 01:01:36.331734 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.329324    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-credential-socket\") pod \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") "
Nov 03 01:01:36.331734 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.329423    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-token\" (UniqueName: \"kubernetes.io/projected/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-token\") pod \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") "
Nov 03 01:01:36.331734 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.329538    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e95dc58-cf96-48fd-91b0-7ce6156ed573-kube-api-access-s6885\" (UniqueName: \"kubernetes.io/projected/8e95dc58-cf96-48fd-91b0-7ce6156ed573-kube-api-access-s6885\") pod \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") "
Nov 03 01:01:36.331734 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.329566    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-data\") pod \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") "
Nov 03 01:01:36.331734 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.329646    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-workload-certs\") pod \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") "
Nov 03 01:01:36.331734 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.329720    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istiod-ca-cert\") pod \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\" (UID: \"8e95dc58-cf96-48fd-91b0-7ce6156ed573\") "
Nov 03 01:01:36.356490 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.347704    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-workload-certs" pod "8e95dc58-cf96-48fd-91b0-7ce6156ed573" (UID: "8e95dc58-cf96-48fd-91b0-7ce6156ed573"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:01:36.356490 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.348147    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istiod-ca-cert" pod "8e95dc58-cf96-48fd-91b0-7ce6156ed573" (UID: "8e95dc58-cf96-48fd-91b0-7ce6156ed573"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:01:36.366309 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.365204    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-credential-socket" pod "8e95dc58-cf96-48fd-91b0-7ce6156ed573" (UID: "8e95dc58-cf96-48fd-91b0-7ce6156ed573"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:01:36.371854 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.371811    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-workload-socket" pod "8e95dc58-cf96-48fd-91b0-7ce6156ed573" (UID: "8e95dc58-cf96-48fd-91b0-7ce6156ed573"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:01:36.372318 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.372284    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-data" pod "8e95dc58-cf96-48fd-91b0-7ce6156ed573" (UID: "8e95dc58-cf96-48fd-91b0-7ce6156ed573"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:01:36.381403 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.378265    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-podinfo" pod "8e95dc58-cf96-48fd-91b0-7ce6156ed573" (UID: "8e95dc58-cf96-48fd-91b0-7ce6156ed573"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:01:36.381403 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.378291    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e95dc58-cf96-48fd-91b0-7ce6156ed573-kube-api-access-s6885" pod "8e95dc58-cf96-48fd-91b0-7ce6156ed573" (UID: "8e95dc58-cf96-48fd-91b0-7ce6156ed573"). InnerVolumeSpecName "kube-api-access-s6885". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:01:36.381403 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.378650    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-token" pod "8e95dc58-cf96-48fd-91b0-7ce6156ed573" (UID: "8e95dc58-cf96-48fd-91b0-7ce6156ed573"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:01:36.381403 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.380301    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-envoy" pod "8e95dc58-cf96-48fd-91b0-7ce6156ed573" (UID: "8e95dc58-cf96-48fd-91b0-7ce6156ed573"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:01:36.432742 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.432717    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:01:36.432742 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.432742    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:01:36.432904 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.432751    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:01:36.432904 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.432761    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:01:36.432904 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.432770    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:01:36.432904 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.432778    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:01:36.432904 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.432786    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6885\" (UniqueName: \"kubernetes.io/projected/8e95dc58-cf96-48fd-91b0-7ce6156ed573-kube-api-access-s6885\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:01:36.432904 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.432795    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:01:36.432904 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.432803    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e95dc58-cf96-48fd-91b0-7ce6156ed573-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:01:36.660390 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.660362    5012 generic.go:356] "Generic (PLEG): container finished" podID="8e95dc58-cf96-48fd-91b0-7ce6156ed573" containerID="61bc9e3fc1f70ee519b5f6fd6a231acaa578cc5e69dd2169ca66080cd07dd290" exitCode=0
Nov 03 01:01:36.660551 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.660412    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"
Nov 03 01:01:36.660551 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.660443    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2" event={"ID":"8e95dc58-cf96-48fd-91b0-7ce6156ed573","Type":"ContainerDied","Data":"61bc9e3fc1f70ee519b5f6fd6a231acaa578cc5e69dd2169ca66080cd07dd290"}
Nov 03 01:01:36.660551 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.660496    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2" event={"ID":"8e95dc58-cf96-48fd-91b0-7ce6156ed573","Type":"ContainerDied","Data":"0d6c118769bf95a0b1bd50800f2f11fa74df48c2d0df7d3c850d0c569c62207b"}
Nov 03 01:01:36.660551 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.660519    5012 scope.go:122] "RemoveContainer" containerID="61bc9e3fc1f70ee519b5f6fd6a231acaa578cc5e69dd2169ca66080cd07dd290"
Nov 03 01:01:36.672496 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.672479    5012 scope.go:122] "RemoveContainer" containerID="e3e6fa7439f7bcdc58bae33503ffb76ffc5e00cf45aefd32267ee44e7c1cbb7d"
Nov 03 01:01:36.683128 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.683107    5012 scope.go:122] "RemoveContainer" containerID="61bc9e3fc1f70ee519b5f6fd6a231acaa578cc5e69dd2169ca66080cd07dd290"
Nov 03 01:01:36.683373 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:01:36.683352    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61bc9e3fc1f70ee519b5f6fd6a231acaa578cc5e69dd2169ca66080cd07dd290\": container with ID starting with 61bc9e3fc1f70ee519b5f6fd6a231acaa578cc5e69dd2169ca66080cd07dd290 not found: ID does not exist" containerID="61bc9e3fc1f70ee519b5f6fd6a231acaa578cc5e69dd2169ca66080cd07dd290"
Nov 03 01:01:36.683421 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.683381    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61bc9e3fc1f70ee519b5f6fd6a231acaa578cc5e69dd2169ca66080cd07dd290"} err="failed to get container status \"61bc9e3fc1f70ee519b5f6fd6a231acaa578cc5e69dd2169ca66080cd07dd290\": rpc error: code = NotFound desc = could not find container \"61bc9e3fc1f70ee519b5f6fd6a231acaa578cc5e69dd2169ca66080cd07dd290\": container with ID starting with 61bc9e3fc1f70ee519b5f6fd6a231acaa578cc5e69dd2169ca66080cd07dd290 not found: ID does not exist"
Nov 03 01:01:36.683421 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.683398    5012 scope.go:122] "RemoveContainer" containerID="e3e6fa7439f7bcdc58bae33503ffb76ffc5e00cf45aefd32267ee44e7c1cbb7d"
Nov 03 01:01:36.683622 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:01:36.683603    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3e6fa7439f7bcdc58bae33503ffb76ffc5e00cf45aefd32267ee44e7c1cbb7d\": container with ID starting with e3e6fa7439f7bcdc58bae33503ffb76ffc5e00cf45aefd32267ee44e7c1cbb7d not found: ID does not exist" containerID="e3e6fa7439f7bcdc58bae33503ffb76ffc5e00cf45aefd32267ee44e7c1cbb7d"
Nov 03 01:01:36.683660 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.683628    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3e6fa7439f7bcdc58bae33503ffb76ffc5e00cf45aefd32267ee44e7c1cbb7d"} err="failed to get container status \"e3e6fa7439f7bcdc58bae33503ffb76ffc5e00cf45aefd32267ee44e7c1cbb7d\": rpc error: code = NotFound desc = could not find container \"e3e6fa7439f7bcdc58bae33503ffb76ffc5e00cf45aefd32267ee44e7c1cbb7d\": container with ID starting with e3e6fa7439f7bcdc58bae33503ffb76ffc5e00cf45aefd32267ee44e7c1cbb7d not found: ID does not exist"
Nov 03 01:01:36.709702 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.709680    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"]
Nov 03 01:01:36.720117 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:36.720095    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-4swl2"]
Nov 03 01:01:37.246203 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:37.246129    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e95dc58-cf96-48fd-91b0-7ce6156ed573" path="/var/lib/kubelet/pods/8e95dc58-cf96-48fd-91b0-7ce6156ed573/volumes"
Nov 03 01:01:37.364659 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:01:37.364620    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:01:52.887761 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.887727    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"]
Nov 03 01:01:52.888547 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.888527    5012 cpu_manager.go:414] "containerMap: removing container" podUID="8e95dc58-cf96-48fd-91b0-7ce6156ed573" containerName="istio-proxy"
Nov 03 01:01:52.888547 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.888546    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e95dc58-cf96-48fd-91b0-7ce6156ed573" containerName="istio-proxy"
Nov 03 01:01:52.888673 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.888573    5012 cpu_manager.go:414] "containerMap: removing container" podUID="8e3746c4-7a9a-470e-a4f2-4b48f8863080" containerName="istio-proxy"
Nov 03 01:01:52.888673 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.888578    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e3746c4-7a9a-470e-a4f2-4b48f8863080" containerName="istio-proxy"
Nov 03 01:01:52.888673 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.888595    5012 cpu_manager.go:414] "containerMap: removing container" podUID="f1faca4e-fef0-4781-a61b-52d107115094" containerName="cleanup"
Nov 03 01:01:52.888673 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.888601    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1faca4e-fef0-4781-a61b-52d107115094" containerName="cleanup"
Nov 03 01:01:52.888673 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.888618    5012 cpu_manager.go:414] "containerMap: removing container" podUID="8e95dc58-cf96-48fd-91b0-7ce6156ed573" containerName="istio-proxy"
Nov 03 01:01:52.888673 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.888622    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e95dc58-cf96-48fd-91b0-7ce6156ed573" containerName="istio-proxy"
Nov 03 01:01:52.888673 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.888634    5012 cpu_manager.go:414] "containerMap: removing container" podUID="6b934863-00e3-4318-9d21-8bb7b64d9761" containerName="cleanup"
Nov 03 01:01:52.888673 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.888638    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b934863-00e3-4318-9d21-8bb7b64d9761" containerName="cleanup"
Nov 03 01:01:52.888673 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.888648    5012 cpu_manager.go:414] "containerMap: removing container" podUID="703067f2-bbcf-475f-8ed2-7146f5ad5f6b" containerName="cleanup"
Nov 03 01:01:52.888673 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.888653    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="703067f2-bbcf-475f-8ed2-7146f5ad5f6b" containerName="cleanup"
Nov 03 01:01:52.889336 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.888765    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e95dc58-cf96-48fd-91b0-7ce6156ed573" containerName="istio-proxy"
Nov 03 01:01:52.889336 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.888780    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e95dc58-cf96-48fd-91b0-7ce6156ed573" containerName="istio-proxy"
Nov 03 01:01:52.889336 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.888786    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b934863-00e3-4318-9d21-8bb7b64d9761" containerName="cleanup"
Nov 03 01:01:52.889336 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.888793    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="703067f2-bbcf-475f-8ed2-7146f5ad5f6b" containerName="cleanup"
Nov 03 01:01:52.889336 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.888800    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e3746c4-7a9a-470e-a4f2-4b48f8863080" containerName="istio-proxy"
Nov 03 01:01:52.889336 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.888808    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eea71f4-f5d2-4530-9105-8e4e4c77c4fd" containerName="cleanup"
Nov 03 01:01:52.892905 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.892882    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:01:52.912570 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.912534    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"]
Nov 03 01:01:52.914805 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.914783    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-29d09b\"/\"default-dockercfg-4k6bl\""
Nov 03 01:01:52.915313 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.915135    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-29d09b\"/\"test-model-29d09b-kserve-self-signed-certs\""
Nov 03 01:01:52.919751 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.919730    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-29d09b\"/\"kube-root-ca.crt\""
Nov 03 01:01:52.920022 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.919999    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-29d09b\"/\"openshift-service-ca.crt\""
Nov 03 01:01:52.992690 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.992657    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-home\") pod \"test-model-29d09b-kserve-76cd55f567-qkh57\" (UID: \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\") " pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:01:52.992853 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.992791    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwqxv\" (UniqueName: \"kubernetes.io/projected/b8cf1798-5790-4367-aa58-d8bc7ecc0932-kube-api-access-hwqxv\") pod \"test-model-29d09b-kserve-76cd55f567-qkh57\" (UID: \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\") " pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:01:52.992967 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.992892    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8cf1798-5790-4367-aa58-d8bc7ecc0932-tls-certs\") pod \"test-model-29d09b-kserve-76cd55f567-qkh57\" (UID: \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\") " pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:01:52.993046 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.993025    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-tmp-dir\") pod \"test-model-29d09b-kserve-76cd55f567-qkh57\" (UID: \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\") " pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:01:52.993095 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.993062    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-dshm\") pod \"test-model-29d09b-kserve-76cd55f567-qkh57\" (UID: \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\") " pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:01:52.993141 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:52.993130    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-model-cache\") pod \"test-model-29d09b-kserve-76cd55f567-qkh57\" (UID: \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\") " pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:01:53.094427 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:53.094392    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-home\") pod \"test-model-29d09b-kserve-76cd55f567-qkh57\" (UID: \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\") " pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:01:53.094590 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:53.094499    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwqxv\" (UniqueName: \"kubernetes.io/projected/b8cf1798-5790-4367-aa58-d8bc7ecc0932-kube-api-access-hwqxv\") pod \"test-model-29d09b-kserve-76cd55f567-qkh57\" (UID: \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\") " pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:01:53.094590 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:53.094546    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8cf1798-5790-4367-aa58-d8bc7ecc0932-tls-certs\") pod \"test-model-29d09b-kserve-76cd55f567-qkh57\" (UID: \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\") " pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:01:53.094728 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:53.094594    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-tmp-dir\") pod \"test-model-29d09b-kserve-76cd55f567-qkh57\" (UID: \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\") " pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:01:53.094728 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:53.094616    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-dshm\") pod \"test-model-29d09b-kserve-76cd55f567-qkh57\" (UID: \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\") " pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:01:53.094728 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:53.094665    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-model-cache\") pod \"test-model-29d09b-kserve-76cd55f567-qkh57\" (UID: \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\") " pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:01:53.095005 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:53.094943    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-home\") pod \"test-model-29d09b-kserve-76cd55f567-qkh57\" (UID: \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\") " pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:01:53.095049 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:53.095019    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-tmp-dir\") pod \"test-model-29d09b-kserve-76cd55f567-qkh57\" (UID: \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\") " pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:01:53.095114 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:53.095100    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-model-cache\") pod \"test-model-29d09b-kserve-76cd55f567-qkh57\" (UID: \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\") " pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:01:53.096919 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:53.096885    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-dshm\") pod \"test-model-29d09b-kserve-76cd55f567-qkh57\" (UID: \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\") " pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:01:53.097110 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:53.097092    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8cf1798-5790-4367-aa58-d8bc7ecc0932-tls-certs\") pod \"test-model-29d09b-kserve-76cd55f567-qkh57\" (UID: \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\") " pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:01:53.112299 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:53.112279    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwqxv\" (UniqueName: \"kubernetes.io/projected/b8cf1798-5790-4367-aa58-d8bc7ecc0932-kube-api-access-hwqxv\") pod \"test-model-29d09b-kserve-76cd55f567-qkh57\" (UID: \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\") " pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:01:53.210982 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:53.209678    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:01:53.533502 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:01:53.533471    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8cf1798_5790_4367_aa58_d8bc7ecc0932.slice/crio-7d02c08d0ed63d1fd2d57c9bb0a098de9e5d9b7edccccca0a143e64463e8db6d WatchSource:0}: Error finding container 7d02c08d0ed63d1fd2d57c9bb0a098de9e5d9b7edccccca0a143e64463e8db6d: Status 404 returned error can't find the container with id 7d02c08d0ed63d1fd2d57c9bb0a098de9e5d9b7edccccca0a143e64463e8db6d
Nov 03 01:01:53.533639 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:53.533620    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"]
Nov 03 01:01:53.837312 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:53.837206    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57" event={"ID":"b8cf1798-5790-4367-aa58-d8bc7ecc0932","Type":"ContainerStarted","Data":"0d8baa32d49b8ef323c3466c7f4543717c027a351c17d9d0eff2d592ab607fd3"}
Nov 03 01:01:53.837312 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:53.837264    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57" event={"ID":"b8cf1798-5790-4367-aa58-d8bc7ecc0932","Type":"ContainerStarted","Data":"7d02c08d0ed63d1fd2d57c9bb0a098de9e5d9b7edccccca0a143e64463e8db6d"}
Nov 03 01:01:53.877628 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:53.877585    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57" podStartSLOduration=1.877573514 podStartE2EDuration="1.877573514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:53.872612447 +0000 UTC m=+3396.752196493" watchObservedRunningTime="2026-11-03 01:01:53.877573514 +0000 UTC m=+3396.757157557"
Nov 03 01:01:54.210453 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:01:54.210420    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:02:03.210263 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:03.210225    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:02:03.226882 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:03.226311    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:02:03.964162 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:03.964134    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:02:07.365645 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:02:07.365604    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:02:16.672016 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.671972    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"]
Nov 03 01:02:16.679659 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.679628    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:02:16.698585 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.698553    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"]
Nov 03 01:02:16.703021 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.702997    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-537652\"/\"default-dockercfg-tll8w\""
Nov 03 01:02:16.703170 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.703063    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-537652\"/\"openshift-service-ca.crt\""
Nov 03 01:02:16.703170 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.703082    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-537652\"/\"kube-root-ca.crt\""
Nov 03 01:02:16.703170 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.703113    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-537652\"/\"test-model-537652-kserve-self-signed-certs\""
Nov 03 01:02:16.746076 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.746034    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-tmp-dir\") pod \"test-model-537652-kserve-78755f497f-p65pc\" (UID: \"5d60e1f1-f8dc-4863-a9db-49face095aa9\") " pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:02:16.746286 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.746097    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhlrb\" (UniqueName: \"kubernetes.io/projected/5d60e1f1-f8dc-4863-a9db-49face095aa9-kube-api-access-hhlrb\") pod \"test-model-537652-kserve-78755f497f-p65pc\" (UID: \"5d60e1f1-f8dc-4863-a9db-49face095aa9\") " pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:02:16.746286 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.746240    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-model-cache\") pod \"test-model-537652-kserve-78755f497f-p65pc\" (UID: \"5d60e1f1-f8dc-4863-a9db-49face095aa9\") " pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:02:16.746410 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.746294    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-home\") pod \"test-model-537652-kserve-78755f497f-p65pc\" (UID: \"5d60e1f1-f8dc-4863-a9db-49face095aa9\") " pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:02:16.746466 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.746436    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d60e1f1-f8dc-4863-a9db-49face095aa9-tls-certs\") pod \"test-model-537652-kserve-78755f497f-p65pc\" (UID: \"5d60e1f1-f8dc-4863-a9db-49face095aa9\") " pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:02:16.746739 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.746715    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-dshm\") pod \"test-model-537652-kserve-78755f497f-p65pc\" (UID: \"5d60e1f1-f8dc-4863-a9db-49face095aa9\") " pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:02:16.847876 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.847841    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-dshm\") pod \"test-model-537652-kserve-78755f497f-p65pc\" (UID: \"5d60e1f1-f8dc-4863-a9db-49face095aa9\") " pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:02:16.848111 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.847939    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-tmp-dir\") pod \"test-model-537652-kserve-78755f497f-p65pc\" (UID: \"5d60e1f1-f8dc-4863-a9db-49face095aa9\") " pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:02:16.848111 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.847974    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhlrb\" (UniqueName: \"kubernetes.io/projected/5d60e1f1-f8dc-4863-a9db-49face095aa9-kube-api-access-hhlrb\") pod \"test-model-537652-kserve-78755f497f-p65pc\" (UID: \"5d60e1f1-f8dc-4863-a9db-49face095aa9\") " pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:02:16.848111 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.848052    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-model-cache\") pod \"test-model-537652-kserve-78755f497f-p65pc\" (UID: \"5d60e1f1-f8dc-4863-a9db-49face095aa9\") " pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:02:16.848111 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.848100    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-home\") pod \"test-model-537652-kserve-78755f497f-p65pc\" (UID: \"5d60e1f1-f8dc-4863-a9db-49face095aa9\") " pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:02:16.848362 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.848159    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d60e1f1-f8dc-4863-a9db-49face095aa9-tls-certs\") pod \"test-model-537652-kserve-78755f497f-p65pc\" (UID: \"5d60e1f1-f8dc-4863-a9db-49face095aa9\") " pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:02:16.848473 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.848449    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-tmp-dir\") pod \"test-model-537652-kserve-78755f497f-p65pc\" (UID: \"5d60e1f1-f8dc-4863-a9db-49face095aa9\") " pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:02:16.849071 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.849041    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-home\") pod \"test-model-537652-kserve-78755f497f-p65pc\" (UID: \"5d60e1f1-f8dc-4863-a9db-49face095aa9\") " pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:02:16.849371 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.849346    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-model-cache\") pod \"test-model-537652-kserve-78755f497f-p65pc\" (UID: \"5d60e1f1-f8dc-4863-a9db-49face095aa9\") " pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:02:16.851885 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.851862    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-dshm\") pod \"test-model-537652-kserve-78755f497f-p65pc\" (UID: \"5d60e1f1-f8dc-4863-a9db-49face095aa9\") " pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:02:16.852610 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.852588    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d60e1f1-f8dc-4863-a9db-49face095aa9-tls-certs\") pod \"test-model-537652-kserve-78755f497f-p65pc\" (UID: \"5d60e1f1-f8dc-4863-a9db-49face095aa9\") " pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:02:16.866040 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.866004    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhlrb\" (UniqueName: \"kubernetes.io/projected/5d60e1f1-f8dc-4863-a9db-49face095aa9-kube-api-access-hhlrb\") pod \"test-model-537652-kserve-78755f497f-p65pc\" (UID: \"5d60e1f1-f8dc-4863-a9db-49face095aa9\") " pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:02:16.999745 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:16.999641    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:02:17.339890 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:02:17.339854    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d60e1f1_f8dc_4863_a9db_49face095aa9.slice/crio-bf9cd8b08b444a124875423d61a81460d884db5bd877aaf635fa7740516d738b WatchSource:0}: Error finding container bf9cd8b08b444a124875423d61a81460d884db5bd877aaf635fa7740516d738b: Status 404 returned error can't find the container with id bf9cd8b08b444a124875423d61a81460d884db5bd877aaf635fa7740516d738b
Nov 03 01:02:17.340529 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:17.340500    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"]
Nov 03 01:02:18.180162 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:18.180122    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc" event={"ID":"5d60e1f1-f8dc-4863-a9db-49face095aa9","Type":"ContainerStarted","Data":"be63964787c57d83f7fe641734981ada866a20cc917518c4bf4e21c57d509315"}
Nov 03 01:02:18.180600 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:18.180184    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc" event={"ID":"5d60e1f1-f8dc-4863-a9db-49face095aa9","Type":"ContainerStarted","Data":"bf9cd8b08b444a124875423d61a81460d884db5bd877aaf635fa7740516d738b"}
Nov 03 01:02:18.218455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:18.218399    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc" podStartSLOduration=2.218384844 podStartE2EDuration="2.218384844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:02:18.215443428 +0000 UTC m=+3421.095027471" watchObservedRunningTime="2026-11-03 01:02:18.218384844 +0000 UTC m=+3421.097968886"
Nov 03 01:02:19.003283 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:19.003244    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:02:25.683577 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.681685    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"]
Nov 03 01:02:25.702011 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.699489    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.702011 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.701557    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"]
Nov 03 01:02:25.842027 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.841981    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.842249 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.842071    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.842355 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.842250    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.842440 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.842358    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.842500 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.842456    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l65zk\" (UniqueName: \"kubernetes.io/projected/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-kube-api-access-l65zk\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.842627 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.842609    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.842690 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.842654    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.842690 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.842678    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.842870 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.842823    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.944526 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.944444    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.944682 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.944553    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.944682 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.944600    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.944682 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.944658    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l65zk\" (UniqueName: \"kubernetes.io/projected/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-kube-api-access-l65zk\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.944829 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.944739    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.944829 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.944767    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.944829 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.944794    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.945020 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.944861    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.945020 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.944884    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.945020 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.944985    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.945020 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.944991    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.945255 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.945127    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.945255 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.945241    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.945561 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.945536    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.947091 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.947070    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.947368 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.947345    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.962079 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.962048    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:25.962217 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:25.962118    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l65zk\" (UniqueName: \"kubernetes.io/projected/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-kube-api-access-l65zk\") pod \"maas-default-gateway-openshift-default-6cc458d887-vrgzs\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:26.059950 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:26.059812    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:26.381861 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:26.381832    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"]
Nov 03 01:02:26.382352 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:02:26.382328    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb260d86b_87b8_4779_b2cb_02c6aa18b9f1.slice/crio-6f1f629b97bb8b3c7443d8d1e97e9236a7afd5ad0fd462a9774f131bad79e8bb WatchSource:0}: Error finding container 6f1f629b97bb8b3c7443d8d1e97e9236a7afd5ad0fd462a9774f131bad79e8bb: Status 404 returned error can't find the container with id 6f1f629b97bb8b3c7443d8d1e97e9236a7afd5ad0fd462a9774f131bad79e8bb
Nov 03 01:02:26.386279 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:26.386237    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:02:26.386360 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:26.386323    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:02:26.386404 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:26.386363    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:02:27.003494 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:27.003455    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:02:27.016845 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:27.016813    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:02:27.268017 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:27.267898    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs" event={"ID":"b260d86b-87b8-4779-b2cb-02c6aa18b9f1","Type":"ContainerStarted","Data":"cb5d68de573cf4e4c8d968c10ce2b548c0c1b9bc68950f0e4ebfd49255773b01"}
Nov 03 01:02:27.268017 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:27.267992    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs" event={"ID":"b260d86b-87b8-4779-b2cb-02c6aa18b9f1","Type":"ContainerStarted","Data":"6f1f629b97bb8b3c7443d8d1e97e9236a7afd5ad0fd462a9774f131bad79e8bb"}
Nov 03 01:02:27.280007 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:27.279964    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:02:27.347530 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:27.347474    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs" podStartSLOduration=2.347457933 podStartE2EDuration="2.347457933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:02:27.308407877 +0000 UTC m=+3430.187991934" watchObservedRunningTime="2026-11-03 01:02:27.347457933 +0000 UTC m=+3430.227041979"
Nov 03 01:02:28.060843 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:28.060811    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:28.063011 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:28.062985    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vrgzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Nov 03 01:02:28.063104 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:28.063034    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs" podUID="b260d86b-87b8-4779-b2cb-02c6aa18b9f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Nov 03 01:02:29.060949 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:29.060871    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vrgzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Nov 03 01:02:29.061385 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:29.060986    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs" podUID="b260d86b-87b8-4779-b2cb-02c6aa18b9f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Nov 03 01:02:30.060549 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:30.060513    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vrgzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Nov 03 01:02:30.060747 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:30.060578    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs" podUID="b260d86b-87b8-4779-b2cb-02c6aa18b9f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Nov 03 01:02:31.060653 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:31.060613    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vrgzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Nov 03 01:02:31.061094 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:31.060668    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs" podUID="b260d86b-87b8-4779-b2cb-02c6aa18b9f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Nov 03 01:02:32.060328 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:32.060295    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vrgzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Nov 03 01:02:32.060510 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:32.060354    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs" podUID="b260d86b-87b8-4779-b2cb-02c6aa18b9f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Nov 03 01:02:33.061118 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:33.061073    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vrgzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Nov 03 01:02:33.061521 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:33.061152    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs" podUID="b260d86b-87b8-4779-b2cb-02c6aa18b9f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Nov 03 01:02:34.060200 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:34.060164    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vrgzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Nov 03 01:02:34.060392 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:34.060231    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs" podUID="b260d86b-87b8-4779-b2cb-02c6aa18b9f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Nov 03 01:02:35.640974 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:35.638474    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:35.640974 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:35.639500    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:35.640974 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:35.640336    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:02:37.371980 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:02:37.371936    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:02:56.664977 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:56.664938    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"]
Nov 03 01:02:56.665562 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:02:56.665153    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k" podUID="209af59a-bfd0-4aab-b2bf-7c789190f57e" containerName="istio-proxy" containerID="cri-o://8c03ea4f5fcb71e7715b3847754388dd36f36878eb7fa0a104e70d6e9a7b73c2" gracePeriod=30
Nov 03 01:03:01.999079 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:01.999049    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 01:03:02.095570 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.095532    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/209af59a-bfd0-4aab-b2bf-7c789190f57e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/209af59a-bfd0-4aab-b2bf-7c789190f57e-istiod-ca-cert\") pod \"209af59a-bfd0-4aab-b2bf-7c789190f57e\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") "
Nov 03 01:03:02.095767 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.095612    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-podinfo\") pod \"209af59a-bfd0-4aab-b2bf-7c789190f57e\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") "
Nov 03 01:03:02.095767 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.095728    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-data\") pod \"209af59a-bfd0-4aab-b2bf-7c789190f57e\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") "
Nov 03 01:03:02.095900 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.095773    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-workload-certs\") pod \"209af59a-bfd0-4aab-b2bf-7c789190f57e\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") "
Nov 03 01:03:02.095900 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.095852    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/209af59a-bfd0-4aab-b2bf-7c789190f57e-kube-api-access-89wxp\" (UniqueName: \"kubernetes.io/projected/209af59a-bfd0-4aab-b2bf-7c789190f57e-kube-api-access-89wxp\") pod \"209af59a-bfd0-4aab-b2bf-7c789190f57e\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") "
Nov 03 01:03:02.096081 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.096050    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-workload-socket\") pod \"209af59a-bfd0-4aab-b2bf-7c789190f57e\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") "
Nov 03 01:03:02.096153 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.096076    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-data" pod "209af59a-bfd0-4aab-b2bf-7c789190f57e" (UID: "209af59a-bfd0-4aab-b2bf-7c789190f57e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:03:02.096153 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.095999    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/209af59a-bfd0-4aab-b2bf-7c789190f57e-istiod-ca-cert" pod "209af59a-bfd0-4aab-b2bf-7c789190f57e" (UID: "209af59a-bfd0-4aab-b2bf-7c789190f57e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:03:02.096153 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.096086    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-envoy\") pod \"209af59a-bfd0-4aab-b2bf-7c789190f57e\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") "
Nov 03 01:03:02.096327 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.096173    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-token\" (UniqueName: \"kubernetes.io/projected/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-token\") pod \"209af59a-bfd0-4aab-b2bf-7c789190f57e\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") "
Nov 03 01:03:02.096327 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.096200    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-workload-certs" pod "209af59a-bfd0-4aab-b2bf-7c789190f57e" (UID: "209af59a-bfd0-4aab-b2bf-7c789190f57e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:03:02.096327 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.096281    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-credential-socket\") pod \"209af59a-bfd0-4aab-b2bf-7c789190f57e\" (UID: \"209af59a-bfd0-4aab-b2bf-7c789190f57e\") "
Nov 03 01:03:02.096498 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.096404    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-workload-socket" pod "209af59a-bfd0-4aab-b2bf-7c789190f57e" (UID: "209af59a-bfd0-4aab-b2bf-7c789190f57e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:03:02.096981 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.096947    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-credential-socket" pod "209af59a-bfd0-4aab-b2bf-7c789190f57e" (UID: "209af59a-bfd0-4aab-b2bf-7c789190f57e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:03:02.097118 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.097051    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/209af59a-bfd0-4aab-b2bf-7c789190f57e-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:03:02.097118 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.097073    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:03:02.097118 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.097086    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:03:02.097118 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.097100    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:03:02.098527 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.098499    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-podinfo" pod "209af59a-bfd0-4aab-b2bf-7c789190f57e" (UID: "209af59a-bfd0-4aab-b2bf-7c789190f57e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:03:02.098658 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.098625    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-envoy" pod "209af59a-bfd0-4aab-b2bf-7c789190f57e" (UID: "209af59a-bfd0-4aab-b2bf-7c789190f57e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:03:02.098788 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.098711    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/209af59a-bfd0-4aab-b2bf-7c789190f57e-kube-api-access-89wxp" pod "209af59a-bfd0-4aab-b2bf-7c789190f57e" (UID: "209af59a-bfd0-4aab-b2bf-7c789190f57e"). InnerVolumeSpecName "kube-api-access-89wxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:03:02.099581 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.099555    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-token" pod "209af59a-bfd0-4aab-b2bf-7c789190f57e" (UID: "209af59a-bfd0-4aab-b2bf-7c789190f57e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:03:02.199079 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.199043    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:03:02.199079 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.199073    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:03:02.199079 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.199084    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/209af59a-bfd0-4aab-b2bf-7c789190f57e-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:03:02.199334 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.199093    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/209af59a-bfd0-4aab-b2bf-7c789190f57e-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:03:02.199334 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.199102    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89wxp\" (UniqueName: \"kubernetes.io/projected/209af59a-bfd0-4aab-b2bf-7c789190f57e-kube-api-access-89wxp\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:03:02.703097 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.703063    5012 generic.go:356] "Generic (PLEG): container finished" podID="209af59a-bfd0-4aab-b2bf-7c789190f57e" containerID="8c03ea4f5fcb71e7715b3847754388dd36f36878eb7fa0a104e70d6e9a7b73c2" exitCode=0
Nov 03 01:03:02.703295 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.703172    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"
Nov 03 01:03:02.703295 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.703209    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k" event={"ID":"209af59a-bfd0-4aab-b2bf-7c789190f57e","Type":"ContainerDied","Data":"8c03ea4f5fcb71e7715b3847754388dd36f36878eb7fa0a104e70d6e9a7b73c2"}
Nov 03 01:03:02.703295 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.703274    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k" event={"ID":"209af59a-bfd0-4aab-b2bf-7c789190f57e","Type":"ContainerDied","Data":"eab23cef46938d1b91b554b57d31912332b7d8ff7db30f52d93b1a092a832694"}
Nov 03 01:03:02.703479 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.703306    5012 scope.go:122] "RemoveContainer" containerID="8c03ea4f5fcb71e7715b3847754388dd36f36878eb7fa0a104e70d6e9a7b73c2"
Nov 03 01:03:02.719066 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.719044    5012 scope.go:122] "RemoveContainer" containerID="5c0755ea0d94d72ad025aa98c515eb99c1a341bd0a7db744155a383853f2ecf8"
Nov 03 01:03:02.730162 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.730139    5012 scope.go:122] "RemoveContainer" containerID="8c03ea4f5fcb71e7715b3847754388dd36f36878eb7fa0a104e70d6e9a7b73c2"
Nov 03 01:03:02.730420 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:03:02.730400    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c03ea4f5fcb71e7715b3847754388dd36f36878eb7fa0a104e70d6e9a7b73c2\": container with ID starting with 8c03ea4f5fcb71e7715b3847754388dd36f36878eb7fa0a104e70d6e9a7b73c2 not found: ID does not exist" containerID="8c03ea4f5fcb71e7715b3847754388dd36f36878eb7fa0a104e70d6e9a7b73c2"
Nov 03 01:03:02.730491 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.730427    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c03ea4f5fcb71e7715b3847754388dd36f36878eb7fa0a104e70d6e9a7b73c2"} err="failed to get container status \"8c03ea4f5fcb71e7715b3847754388dd36f36878eb7fa0a104e70d6e9a7b73c2\": rpc error: code = NotFound desc = could not find container \"8c03ea4f5fcb71e7715b3847754388dd36f36878eb7fa0a104e70d6e9a7b73c2\": container with ID starting with 8c03ea4f5fcb71e7715b3847754388dd36f36878eb7fa0a104e70d6e9a7b73c2 not found: ID does not exist"
Nov 03 01:03:02.730491 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.730444    5012 scope.go:122] "RemoveContainer" containerID="5c0755ea0d94d72ad025aa98c515eb99c1a341bd0a7db744155a383853f2ecf8"
Nov 03 01:03:02.730722 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:03:02.730705    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c0755ea0d94d72ad025aa98c515eb99c1a341bd0a7db744155a383853f2ecf8\": container with ID starting with 5c0755ea0d94d72ad025aa98c515eb99c1a341bd0a7db744155a383853f2ecf8 not found: ID does not exist" containerID="5c0755ea0d94d72ad025aa98c515eb99c1a341bd0a7db744155a383853f2ecf8"
Nov 03 01:03:02.730770 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.730726    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c0755ea0d94d72ad025aa98c515eb99c1a341bd0a7db744155a383853f2ecf8"} err="failed to get container status \"5c0755ea0d94d72ad025aa98c515eb99c1a341bd0a7db744155a383853f2ecf8\": rpc error: code = NotFound desc = could not find container \"5c0755ea0d94d72ad025aa98c515eb99c1a341bd0a7db744155a383853f2ecf8\": container with ID starting with 5c0755ea0d94d72ad025aa98c515eb99c1a341bd0a7db744155a383853f2ecf8 not found: ID does not exist"
Nov 03 01:03:02.767850 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.767825    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"]
Nov 03 01:03:02.778137 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:02.778114    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-c7z9k"]
Nov 03 01:03:03.246390 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:03.246358    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="209af59a-bfd0-4aab-b2bf-7c789190f57e" path="/var/lib/kubelet/pods/209af59a-bfd0-4aab-b2bf-7c789190f57e/volumes"
Nov 03 01:03:07.364152 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:03:07.364115    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:03:37.363509 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:03:37.363468    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:03:55.773141 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:03:55.773093    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"]
Nov 03 01:04:07.372336 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:04:07.372291    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:04:10.802723 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:10.802683    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj" podUID="209e0827-a624-4090-bcc4-4ac72c7eb1e6" containerName="main" containerID="cri-o://23715d6872d34e2e403afd43e060ba71fbdbdf35355c0b597b3b69e11dceb7d2" gracePeriod=45
Nov 03 01:04:11.104380 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.104351    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 01:04:11.143712 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.143682    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-home\" (UniqueName: \"kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-home\") pod \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\" (UID: \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\") "
Nov 03 01:04:11.143832 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.143735    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/209e0827-a624-4090-bcc4-4ac72c7eb1e6-kube-api-access-2hss7\" (UniqueName: \"kubernetes.io/projected/209e0827-a624-4090-bcc4-4ac72c7eb1e6-kube-api-access-2hss7\") pod \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\" (UID: \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\") "
Nov 03 01:04:11.143832 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.143768    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-dshm\") pod \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\" (UID: \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\") "
Nov 03 01:04:11.143832 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.143802    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-tmp-dir\") pod \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\" (UID: \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\") "
Nov 03 01:04:11.144041 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.143993    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-home" pod "209e0827-a624-4090-bcc4-4ac72c7eb1e6" (UID: "209e0827-a624-4090-bcc4-4ac72c7eb1e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:04:11.144041 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.144013    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/209e0827-a624-4090-bcc4-4ac72c7eb1e6-tls-certs\" (UniqueName: \"kubernetes.io/secret/209e0827-a624-4090-bcc4-4ac72c7eb1e6-tls-certs\") pod \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\" (UID: \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\") "
Nov 03 01:04:11.144188 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.144161    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-tmp-dir" pod "209e0827-a624-4090-bcc4-4ac72c7eb1e6" (UID: "209e0827-a624-4090-bcc4-4ac72c7eb1e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:04:11.144254 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.144213    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-model-cache\") pod \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\" (UID: \"209e0827-a624-4090-bcc4-4ac72c7eb1e6\") "
Nov 03 01:04:11.144523 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.144496    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-model-cache" pod "209e0827-a624-4090-bcc4-4ac72c7eb1e6" (UID: "209e0827-a624-4090-bcc4-4ac72c7eb1e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:04:11.144895 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.144871    5012 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-model-cache\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:04:11.145085 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.144900    5012 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-home\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:04:11.145085 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.144980    5012 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-tmp-dir\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:04:11.146302 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.146275    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-dshm" pod "209e0827-a624-4090-bcc4-4ac72c7eb1e6" (UID: "209e0827-a624-4090-bcc4-4ac72c7eb1e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:04:11.146430 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.146401    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/209e0827-a624-4090-bcc4-4ac72c7eb1e6-tls-certs" pod "209e0827-a624-4090-bcc4-4ac72c7eb1e6" (UID: "209e0827-a624-4090-bcc4-4ac72c7eb1e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:04:11.146529 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.146502    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/209e0827-a624-4090-bcc4-4ac72c7eb1e6-kube-api-access-2hss7" pod "209e0827-a624-4090-bcc4-4ac72c7eb1e6" (UID: "209e0827-a624-4090-bcc4-4ac72c7eb1e6"). InnerVolumeSpecName "kube-api-access-2hss7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:04:11.245974 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.245944    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hss7\" (UniqueName: \"kubernetes.io/projected/209e0827-a624-4090-bcc4-4ac72c7eb1e6-kube-api-access-2hss7\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:04:11.246144 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.246126    5012 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/209e0827-a624-4090-bcc4-4ac72c7eb1e6-dshm\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:04:11.246144 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.246141    5012 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/209e0827-a624-4090-bcc4-4ac72c7eb1e6-tls-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:04:11.459750 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.459678    5012 generic.go:356] "Generic (PLEG): container finished" podID="209e0827-a624-4090-bcc4-4ac72c7eb1e6" containerID="23715d6872d34e2e403afd43e060ba71fbdbdf35355c0b597b3b69e11dceb7d2" exitCode=0
Nov 03 01:04:11.460102 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.459936    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj" event={"ID":"209e0827-a624-4090-bcc4-4ac72c7eb1e6","Type":"ContainerDied","Data":"23715d6872d34e2e403afd43e060ba71fbdbdf35355c0b597b3b69e11dceb7d2"}
Nov 03 01:04:11.460102 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.460009    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj" event={"ID":"209e0827-a624-4090-bcc4-4ac72c7eb1e6","Type":"ContainerDied","Data":"e743a2e3c4ab318badd43b68eea586da61081559e318833751372a78a31aefa0"}
Nov 03 01:04:11.460102 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.460007    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"
Nov 03 01:04:11.460363 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.460029    5012 scope.go:122] "RemoveContainer" containerID="23715d6872d34e2e403afd43e060ba71fbdbdf35355c0b597b3b69e11dceb7d2"
Nov 03 01:04:11.483062 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.483030    5012 scope.go:122] "RemoveContainer" containerID="23715d6872d34e2e403afd43e060ba71fbdbdf35355c0b597b3b69e11dceb7d2"
Nov 03 01:04:11.483468 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:04:11.483429    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23715d6872d34e2e403afd43e060ba71fbdbdf35355c0b597b3b69e11dceb7d2\": container with ID starting with 23715d6872d34e2e403afd43e060ba71fbdbdf35355c0b597b3b69e11dceb7d2 not found: ID does not exist" containerID="23715d6872d34e2e403afd43e060ba71fbdbdf35355c0b597b3b69e11dceb7d2"
Nov 03 01:04:11.483550 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.483481    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23715d6872d34e2e403afd43e060ba71fbdbdf35355c0b597b3b69e11dceb7d2"} err="failed to get container status \"23715d6872d34e2e403afd43e060ba71fbdbdf35355c0b597b3b69e11dceb7d2\": rpc error: code = NotFound desc = could not find container \"23715d6872d34e2e403afd43e060ba71fbdbdf35355c0b597b3b69e11dceb7d2\": container with ID starting with 23715d6872d34e2e403afd43e060ba71fbdbdf35355c0b597b3b69e11dceb7d2 not found: ID does not exist"
Nov 03 01:04:11.511685 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.511640    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"]
Nov 03 01:04:11.516654 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:11.516618    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-883a9e/test-model-883a9e-kserve-86d6f745dc-xs7fj"]
Nov 03 01:04:13.248248 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:13.248209    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="209e0827-a624-4090-bcc4-4ac72c7eb1e6" path="/var/lib/kubelet/pods/209e0827-a624-4090-bcc4-4ac72c7eb1e6/volumes"
Nov 03 01:04:24.573395 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:24.573358    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9"]
Nov 03 01:04:24.603042 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:24.603007    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq"]
Nov 03 01:04:29.573676 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:29.573593    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9" podUID="8216aa2d-f296-4d63-ad6b-62e2aa583c5c" containerName="payload-processing" containerID="cri-o://2c43ee91523985f544219addd6bb267746c4c47490909c8c7d43af93b8d35506" gracePeriod=25
Nov 03 01:04:29.603298 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:29.603265    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq" podUID="b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3" containerName="payload-pre-processing" containerID="cri-o://21bdf632fdeb4c117c30a4003951d9b4c8634ba06005d4694f3d7643c2685f94" gracePeriod=25
Nov 03 01:04:29.861824 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:29.861797    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9"
Nov 03 01:04:29.964153 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:29.964116    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8216aa2d-f296-4d63-ad6b-62e2aa583c5c-kube-api-access-kjfjb\" (UniqueName: \"kubernetes.io/projected/8216aa2d-f296-4d63-ad6b-62e2aa583c5c-kube-api-access-kjfjb\") pod \"8216aa2d-f296-4d63-ad6b-62e2aa583c5c\" (UID: \"8216aa2d-f296-4d63-ad6b-62e2aa583c5c\") "
Nov 03 01:04:29.964376 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:29.964364    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8216aa2d-f296-4d63-ad6b-62e2aa583c5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8216aa2d-f296-4d63-ad6b-62e2aa583c5c-plugins-config-volume\") pod \"8216aa2d-f296-4d63-ad6b-62e2aa583c5c\" (UID: \"8216aa2d-f296-4d63-ad6b-62e2aa583c5c\") "
Nov 03 01:04:29.964698 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:29.964666    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8216aa2d-f296-4d63-ad6b-62e2aa583c5c-plugins-config-volume" pod "8216aa2d-f296-4d63-ad6b-62e2aa583c5c" (UID: "8216aa2d-f296-4d63-ad6b-62e2aa583c5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:04:29.964955 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:29.964938    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8216aa2d-f296-4d63-ad6b-62e2aa583c5c-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:04:29.966300 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:29.966271    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8216aa2d-f296-4d63-ad6b-62e2aa583c5c-kube-api-access-kjfjb" pod "8216aa2d-f296-4d63-ad6b-62e2aa583c5c" (UID: "8216aa2d-f296-4d63-ad6b-62e2aa583c5c"). InnerVolumeSpecName "kube-api-access-kjfjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:04:30.066725 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:30.066698    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjfjb\" (UniqueName: \"kubernetes.io/projected/8216aa2d-f296-4d63-ad6b-62e2aa583c5c-kube-api-access-kjfjb\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:04:30.632980 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:30.632942    5012 generic.go:356] "Generic (PLEG): container finished" podID="8216aa2d-f296-4d63-ad6b-62e2aa583c5c" containerID="2c43ee91523985f544219addd6bb267746c4c47490909c8c7d43af93b8d35506" exitCode=0
Nov 03 01:04:30.633422 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:30.632987    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9"
Nov 03 01:04:30.633422 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:30.633021    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9" event={"ID":"8216aa2d-f296-4d63-ad6b-62e2aa583c5c","Type":"ContainerDied","Data":"2c43ee91523985f544219addd6bb267746c4c47490909c8c7d43af93b8d35506"}
Nov 03 01:04:30.633422 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:30.633077    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9" event={"ID":"8216aa2d-f296-4d63-ad6b-62e2aa583c5c","Type":"ContainerDied","Data":"6000608dd5131cc76ca234bd21104d9a5dbf5267f23b69d1b9b512659c4ccaad"}
Nov 03 01:04:30.633422 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:30.633099    5012 scope.go:122] "RemoveContainer" containerID="2c43ee91523985f544219addd6bb267746c4c47490909c8c7d43af93b8d35506"
Nov 03 01:04:30.647535 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:30.647511    5012 scope.go:122] "RemoveContainer" containerID="2c43ee91523985f544219addd6bb267746c4c47490909c8c7d43af93b8d35506"
Nov 03 01:04:30.647854 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:04:30.647831    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c43ee91523985f544219addd6bb267746c4c47490909c8c7d43af93b8d35506\": container with ID starting with 2c43ee91523985f544219addd6bb267746c4c47490909c8c7d43af93b8d35506 not found: ID does not exist" containerID="2c43ee91523985f544219addd6bb267746c4c47490909c8c7d43af93b8d35506"
Nov 03 01:04:30.647935 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:30.647861    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c43ee91523985f544219addd6bb267746c4c47490909c8c7d43af93b8d35506"} err="failed to get container status \"2c43ee91523985f544219addd6bb267746c4c47490909c8c7d43af93b8d35506\": rpc error: code = NotFound desc = could not find container \"2c43ee91523985f544219addd6bb267746c4c47490909c8c7d43af93b8d35506\": container with ID starting with 2c43ee91523985f544219addd6bb267746c4c47490909c8c7d43af93b8d35506 not found: ID does not exist"
Nov 03 01:04:30.681093 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:30.681070    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9"]
Nov 03 01:04:30.691383 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:30.691360    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-883a9e-545cdf9bd9-kf4t9"]
Nov 03 01:04:31.246136 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:31.246102    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8216aa2d-f296-4d63-ad6b-62e2aa583c5c" path="/var/lib/kubelet/pods/8216aa2d-f296-4d63-ad6b-62e2aa583c5c/volumes"
Nov 03 01:04:36.056363 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:36.056319    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq" podUID="b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:04:37.372004 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:04:37.371962    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:04:46.056030 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:46.055987    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq" podUID="b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:04:49.849081 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:49.848881    5012 generic.go:356] "Generic (PLEG): container finished" podID="b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3" containerID="21bdf632fdeb4c117c30a4003951d9b4c8634ba06005d4694f3d7643c2685f94" exitCode=0
Nov 03 01:04:49.849081 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:49.848943    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq" event={"ID":"b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3","Type":"ContainerDied","Data":"21bdf632fdeb4c117c30a4003951d9b4c8634ba06005d4694f3d7643c2685f94"}
Nov 03 01:04:49.902608 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:49.902584    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq"
Nov 03 01:04:50.002573 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:50.002524    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3-kube-api-access-twqdd\" (UniqueName: \"kubernetes.io/projected/b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3-kube-api-access-twqdd\") pod \"b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3\" (UID: \"b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3\") "
Nov 03 01:04:50.002886 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:50.002858    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3-plugins-config-volume\") pod \"b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3\" (UID: \"b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3\") "
Nov 03 01:04:50.003338 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:50.003309    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3-plugins-config-volume" pod "b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3" (UID: "b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:04:50.003495 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:50.003479    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:04:50.004994 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:50.004968    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3-kube-api-access-twqdd" pod "b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3" (UID: "b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3"). InnerVolumeSpecName "kube-api-access-twqdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:04:50.105935 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:50.105494    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twqdd\" (UniqueName: \"kubernetes.io/projected/b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3-kube-api-access-twqdd\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:04:50.861369 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:50.861342    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq"
Nov 03 01:04:50.861772 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:50.861341    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq" event={"ID":"b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3","Type":"ContainerDied","Data":"6ff8e01204bdd345e19ca6ce04d6f2f50e188975e70a11ebe84c6b22207c4095"}
Nov 03 01:04:50.861772 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:50.861477    5012 scope.go:122] "RemoveContainer" containerID="21bdf632fdeb4c117c30a4003951d9b4c8634ba06005d4694f3d7643c2685f94"
Nov 03 01:04:50.912428 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:50.912403    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq"]
Nov 03 01:04:50.922219 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:50.922174    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-883a9e-6f759c5bc5-8cldq"]
Nov 03 01:04:51.245722 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:51.245651    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3" path="/var/lib/kubelet/pods/b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3/volumes"
Nov 03 01:04:54.917396 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:04:54.917358    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"]
Nov 03 01:05:05.027264 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:05.027225    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2c2816f-5dfxh"]
Nov 03 01:05:05.037423 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:05.037397    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2c2816f-5dfxh"]
Nov 03 01:05:05.246180 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:05.246149    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="824e8c00-991b-43a9-a368-04b7baa1e746" path="/var/lib/kubelet/pods/824e8c00-991b-43a9-a368-04b7baa1e746/volumes"
Nov 03 01:05:07.365357 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:05:07.365321    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:05:08.835433 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:08.835397    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7"]
Nov 03 01:05:08.836574 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:08.835674    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7" podUID="77f5e43c-70c1-4f1b-ac62-412c0939ecbc" containerName="maas-api" containerID="cri-o://3a48527205c0f9c13e9d7eb7f6b0307d0e07e3883841f60b7a465938670e14c8" gracePeriod=30
Nov 03 01:05:09.542841 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:09.542811    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7"
Nov 03 01:05:09.633466 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:09.633434    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-metrics-certs\") pod \"77f5e43c-70c1-4f1b-ac62-412c0939ecbc\" (UID: \"77f5e43c-70c1-4f1b-ac62-412c0939ecbc\") "
Nov 03 01:05:09.633679 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:09.633615    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-kube-api-access-kxk99\" (UniqueName: \"kubernetes.io/projected/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-kube-api-access-kxk99\") pod \"77f5e43c-70c1-4f1b-ac62-412c0939ecbc\" (UID: \"77f5e43c-70c1-4f1b-ac62-412c0939ecbc\") "
Nov 03 01:05:09.633763 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:09.633702    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-maas-api-tls\") pod \"77f5e43c-70c1-4f1b-ac62-412c0939ecbc\" (UID: \"77f5e43c-70c1-4f1b-ac62-412c0939ecbc\") "
Nov 03 01:05:09.634080 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:09.634048    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-openshift-service-ca\") pod \"77f5e43c-70c1-4f1b-ac62-412c0939ecbc\" (UID: \"77f5e43c-70c1-4f1b-ac62-412c0939ecbc\") "
Nov 03 01:05:09.634388 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:09.634359    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-openshift-service-ca" pod "77f5e43c-70c1-4f1b-ac62-412c0939ecbc" (UID: "77f5e43c-70c1-4f1b-ac62-412c0939ecbc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:05:09.634941 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:09.634900    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:05:09.636351 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:09.636325    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-metrics-certs" pod "77f5e43c-70c1-4f1b-ac62-412c0939ecbc" (UID: "77f5e43c-70c1-4f1b-ac62-412c0939ecbc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:05:09.638304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:09.638278    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-maas-api-tls" pod "77f5e43c-70c1-4f1b-ac62-412c0939ecbc" (UID: "77f5e43c-70c1-4f1b-ac62-412c0939ecbc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:05:09.638439 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:09.638321    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-kube-api-access-kxk99" pod "77f5e43c-70c1-4f1b-ac62-412c0939ecbc" (UID: "77f5e43c-70c1-4f1b-ac62-412c0939ecbc"). InnerVolumeSpecName "kube-api-access-kxk99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:05:09.758066 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:09.752992    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:05:09.758066 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:09.753033    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxk99\" (UniqueName: \"kubernetes.io/projected/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-kube-api-access-kxk99\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:05:09.758066 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:09.753052    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77f5e43c-70c1-4f1b-ac62-412c0939ecbc-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:05:09.941666 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:09.941626    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp" podUID="72d32ad0-f7d6-4c17-8781-83d65335f0e6" containerName="main" containerID="cri-o://930139bcac57df84bddd7baf280b712e14ea28daf3a48cdbaa01e10b7d8b1132" gracePeriod=45
Nov 03 01:05:10.035567 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.035488    5012 generic.go:356] "Generic (PLEG): container finished" podID="77f5e43c-70c1-4f1b-ac62-412c0939ecbc" containerID="3a48527205c0f9c13e9d7eb7f6b0307d0e07e3883841f60b7a465938670e14c8" exitCode=0
Nov 03 01:05:10.035567 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.035538    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7"
Nov 03 01:05:10.035761 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.035535    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7" event={"ID":"77f5e43c-70c1-4f1b-ac62-412c0939ecbc","Type":"ContainerDied","Data":"3a48527205c0f9c13e9d7eb7f6b0307d0e07e3883841f60b7a465938670e14c8"}
Nov 03 01:05:10.035761 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.035641    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7" event={"ID":"77f5e43c-70c1-4f1b-ac62-412c0939ecbc","Type":"ContainerDied","Data":"20f34d2628022a653b00d29f186b0de7d3c44076cbbd3cb30e48f4116a5458f0"}
Nov 03 01:05:10.035761 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.035688    5012 scope.go:122] "RemoveContainer" containerID="3a48527205c0f9c13e9d7eb7f6b0307d0e07e3883841f60b7a465938670e14c8"
Nov 03 01:05:10.055135 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.055115    5012 scope.go:122] "RemoveContainer" containerID="3a48527205c0f9c13e9d7eb7f6b0307d0e07e3883841f60b7a465938670e14c8"
Nov 03 01:05:10.055405 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:05:10.055385    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a48527205c0f9c13e9d7eb7f6b0307d0e07e3883841f60b7a465938670e14c8\": container with ID starting with 3a48527205c0f9c13e9d7eb7f6b0307d0e07e3883841f60b7a465938670e14c8 not found: ID does not exist" containerID="3a48527205c0f9c13e9d7eb7f6b0307d0e07e3883841f60b7a465938670e14c8"
Nov 03 01:05:10.055458 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.055416    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a48527205c0f9c13e9d7eb7f6b0307d0e07e3883841f60b7a465938670e14c8"} err="failed to get container status \"3a48527205c0f9c13e9d7eb7f6b0307d0e07e3883841f60b7a465938670e14c8\": rpc error: code = NotFound desc = could not find container \"3a48527205c0f9c13e9d7eb7f6b0307d0e07e3883841f60b7a465938670e14c8\": container with ID starting with 3a48527205c0f9c13e9d7eb7f6b0307d0e07e3883841f60b7a465938670e14c8 not found: ID does not exist"
Nov 03 01:05:10.083965 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.083938    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7"]
Nov 03 01:05:10.092940 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.092903    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49d7e5-656c7498f8-rj8k7"]
Nov 03 01:05:10.227369 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.227344    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 01:05:10.259991 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.259963    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72d32ad0-f7d6-4c17-8781-83d65335f0e6-tls-certs\" (UniqueName: \"kubernetes.io/secret/72d32ad0-f7d6-4c17-8781-83d65335f0e6-tls-certs\") pod \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\" (UID: \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\") "
Nov 03 01:05:10.260090 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.259998    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-home\" (UniqueName: \"kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-home\") pod \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\" (UID: \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\") "
Nov 03 01:05:10.260163 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.260096    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-tmp-dir\") pod \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\" (UID: \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\") "
Nov 03 01:05:10.260163 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.260133    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-dshm\") pod \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\" (UID: \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\") "
Nov 03 01:05:10.260286 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.260241    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-model-cache\") pod \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\" (UID: \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\") "
Nov 03 01:05:10.260351 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.260331    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-home" pod "72d32ad0-f7d6-4c17-8781-83d65335f0e6" (UID: "72d32ad0-f7d6-4c17-8781-83d65335f0e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:05:10.260409 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.260350    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-tmp-dir" pod "72d32ad0-f7d6-4c17-8781-83d65335f0e6" (UID: "72d32ad0-f7d6-4c17-8781-83d65335f0e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:05:10.260409 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.260380    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72d32ad0-f7d6-4c17-8781-83d65335f0e6-kube-api-access-h7sw7\" (UniqueName: \"kubernetes.io/projected/72d32ad0-f7d6-4c17-8781-83d65335f0e6-kube-api-access-h7sw7\") pod \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\" (UID: \"72d32ad0-f7d6-4c17-8781-83d65335f0e6\") "
Nov 03 01:05:10.260546 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.260523    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-model-cache" pod "72d32ad0-f7d6-4c17-8781-83d65335f0e6" (UID: "72d32ad0-f7d6-4c17-8781-83d65335f0e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:05:10.260872 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.260851    5012 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-model-cache\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:05:10.260872 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.260873    5012 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-home\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:05:10.261090 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.260881    5012 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-tmp-dir\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:05:10.262426 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.262393    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-dshm" pod "72d32ad0-f7d6-4c17-8781-83d65335f0e6" (UID: "72d32ad0-f7d6-4c17-8781-83d65335f0e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:05:10.262541 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.262481    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72d32ad0-f7d6-4c17-8781-83d65335f0e6-tls-certs" pod "72d32ad0-f7d6-4c17-8781-83d65335f0e6" (UID: "72d32ad0-f7d6-4c17-8781-83d65335f0e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:05:10.262541 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.262515    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d32ad0-f7d6-4c17-8781-83d65335f0e6-kube-api-access-h7sw7" pod "72d32ad0-f7d6-4c17-8781-83d65335f0e6" (UID: "72d32ad0-f7d6-4c17-8781-83d65335f0e6"). InnerVolumeSpecName "kube-api-access-h7sw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:05:10.362731 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.362664    5012 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72d32ad0-f7d6-4c17-8781-83d65335f0e6-tls-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:05:10.362731 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.362719    5012 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72d32ad0-f7d6-4c17-8781-83d65335f0e6-dshm\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:05:10.362948 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:10.362744    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7sw7\" (UniqueName: \"kubernetes.io/projected/72d32ad0-f7d6-4c17-8781-83d65335f0e6-kube-api-access-h7sw7\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:05:11.045620 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:11.045575    5012 generic.go:356] "Generic (PLEG): container finished" podID="72d32ad0-f7d6-4c17-8781-83d65335f0e6" containerID="930139bcac57df84bddd7baf280b712e14ea28daf3a48cdbaa01e10b7d8b1132" exitCode=0
Nov 03 01:05:11.046153 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:11.045624    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"
Nov 03 01:05:11.046153 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:11.045631    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp" event={"ID":"72d32ad0-f7d6-4c17-8781-83d65335f0e6","Type":"ContainerDied","Data":"930139bcac57df84bddd7baf280b712e14ea28daf3a48cdbaa01e10b7d8b1132"}
Nov 03 01:05:11.046153 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:11.045690    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp" event={"ID":"72d32ad0-f7d6-4c17-8781-83d65335f0e6","Type":"ContainerDied","Data":"f8c07a8bc0cb6eebc1366f2c20189277ce83e5cf64ab85109a49a877fdd83050"}
Nov 03 01:05:11.046153 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:11.045714    5012 scope.go:122] "RemoveContainer" containerID="930139bcac57df84bddd7baf280b712e14ea28daf3a48cdbaa01e10b7d8b1132"
Nov 03 01:05:11.060577 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:11.060557    5012 scope.go:122] "RemoveContainer" containerID="930139bcac57df84bddd7baf280b712e14ea28daf3a48cdbaa01e10b7d8b1132"
Nov 03 01:05:11.061051 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:05:11.061028    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"930139bcac57df84bddd7baf280b712e14ea28daf3a48cdbaa01e10b7d8b1132\": container with ID starting with 930139bcac57df84bddd7baf280b712e14ea28daf3a48cdbaa01e10b7d8b1132 not found: ID does not exist" containerID="930139bcac57df84bddd7baf280b712e14ea28daf3a48cdbaa01e10b7d8b1132"
Nov 03 01:05:11.061121 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:11.061057    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"930139bcac57df84bddd7baf280b712e14ea28daf3a48cdbaa01e10b7d8b1132"} err="failed to get container status \"930139bcac57df84bddd7baf280b712e14ea28daf3a48cdbaa01e10b7d8b1132\": rpc error: code = NotFound desc = could not find container \"930139bcac57df84bddd7baf280b712e14ea28daf3a48cdbaa01e10b7d8b1132\": container with ID starting with 930139bcac57df84bddd7baf280b712e14ea28daf3a48cdbaa01e10b7d8b1132 not found: ID does not exist"
Nov 03 01:05:11.094232 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:11.094205    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"]
Nov 03 01:05:11.104118 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:11.104095    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7f20f7/test-model-7f20f7-kserve-f656db587-plwfp"]
Nov 03 01:05:11.246288 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:11.246256    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72d32ad0-f7d6-4c17-8781-83d65335f0e6" path="/var/lib/kubelet/pods/72d32ad0-f7d6-4c17-8781-83d65335f0e6/volumes"
Nov 03 01:05:11.246737 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:11.246722    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77f5e43c-70c1-4f1b-ac62-412c0939ecbc" path="/var/lib/kubelet/pods/77f5e43c-70c1-4f1b-ac62-412c0939ecbc/volumes"
Nov 03 01:05:18.504962 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:18.504904    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 01:05:18.508834 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:18.508809    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 01:05:18.509675 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:18.509658    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 01:05:18.513385 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:18.513365    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 01:05:22.061297 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:22.061262    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v"]
Nov 03 01:05:22.061701 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:22.061460    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v" podUID="4d5d2f80-69a4-45d5-90e1-2a8335e38cda" containerName="maas-api" containerID="cri-o://bf08357aeede1fd002cf857c0ac385d4ee09a208d63313a362fa26e8333b5b12" gracePeriod=30
Nov 03 01:05:22.207550 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:22.207517    5012 generic.go:356] "Generic (PLEG): container finished" podID="4d5d2f80-69a4-45d5-90e1-2a8335e38cda" containerID="bf08357aeede1fd002cf857c0ac385d4ee09a208d63313a362fa26e8333b5b12" exitCode=0
Nov 03 01:05:22.207714 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:22.207546    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v" event={"ID":"4d5d2f80-69a4-45d5-90e1-2a8335e38cda","Type":"ContainerDied","Data":"bf08357aeede1fd002cf857c0ac385d4ee09a208d63313a362fa26e8333b5b12"}
Nov 03 01:05:22.482317 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:22.481946    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v"
Nov 03 01:05:22.536960 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:22.532548    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-openshift-service-ca\") pod \"4d5d2f80-69a4-45d5-90e1-2a8335e38cda\" (UID: \"4d5d2f80-69a4-45d5-90e1-2a8335e38cda\") "
Nov 03 01:05:22.536960 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:22.532608    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-metrics-certs\") pod \"4d5d2f80-69a4-45d5-90e1-2a8335e38cda\" (UID: \"4d5d2f80-69a4-45d5-90e1-2a8335e38cda\") "
Nov 03 01:05:22.536960 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:22.532785    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-maas-api-tls\") pod \"4d5d2f80-69a4-45d5-90e1-2a8335e38cda\" (UID: \"4d5d2f80-69a4-45d5-90e1-2a8335e38cda\") "
Nov 03 01:05:22.536960 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:22.532816    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-kube-api-access-6gm5s\" (UniqueName: \"kubernetes.io/projected/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-kube-api-access-6gm5s\") pod \"4d5d2f80-69a4-45d5-90e1-2a8335e38cda\" (UID: \"4d5d2f80-69a4-45d5-90e1-2a8335e38cda\") "
Nov 03 01:05:22.539546 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:22.539495    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-openshift-service-ca" pod "4d5d2f80-69a4-45d5-90e1-2a8335e38cda" (UID: "4d5d2f80-69a4-45d5-90e1-2a8335e38cda"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:05:22.555344 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:22.555275    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-metrics-certs" pod "4d5d2f80-69a4-45d5-90e1-2a8335e38cda" (UID: "4d5d2f80-69a4-45d5-90e1-2a8335e38cda"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:05:22.595159 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:22.595099    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-maas-api-tls" pod "4d5d2f80-69a4-45d5-90e1-2a8335e38cda" (UID: "4d5d2f80-69a4-45d5-90e1-2a8335e38cda"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:05:22.597522 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:22.597484    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-kube-api-access-6gm5s" pod "4d5d2f80-69a4-45d5-90e1-2a8335e38cda" (UID: "4d5d2f80-69a4-45d5-90e1-2a8335e38cda"). InnerVolumeSpecName "kube-api-access-6gm5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:05:22.648477 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:22.648374    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:05:22.648477 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:22.648425    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:05:22.648477 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:22.648440    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:05:22.648477 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:22.648453    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gm5s\" (UniqueName: \"kubernetes.io/projected/4d5d2f80-69a4-45d5-90e1-2a8335e38cda-kube-api-access-6gm5s\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:05:23.218604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:23.218566    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v" event={"ID":"4d5d2f80-69a4-45d5-90e1-2a8335e38cda","Type":"ContainerDied","Data":"a08690fed43237383f48f5658d63195edbca21a928e153690257c7956300472d"}
Nov 03 01:05:23.219034 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:23.218612    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v"
Nov 03 01:05:23.219034 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:23.218651    5012 scope.go:122] "RemoveContainer" containerID="bf08357aeede1fd002cf857c0ac385d4ee09a208d63313a362fa26e8333b5b12"
Nov 03 01:05:23.269938 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:23.269898    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v"]
Nov 03 01:05:23.279938 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:23.279888    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-883a9e-6748f86d8d-ggx9v"]
Nov 03 01:05:25.245117 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:05:25.245083    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d5d2f80-69a4-45d5-90e1-2a8335e38cda" path="/var/lib/kubelet/pods/4d5d2f80-69a4-45d5-90e1-2a8335e38cda/volumes"
Nov 03 01:05:37.366728 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:05:37.366690    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:06:07.365721 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:06:07.365640    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:06:11.630144 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:11.630085    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd"]
Nov 03 01:06:11.631019 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:11.630376    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd" podUID="8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed" containerName="maas-api" containerID="cri-o://842cb3f77adb8df1b2e0fdd10d1467a10e247f7a985e43fb2c7c0f9c7ac7e77e" gracePeriod=30
Nov 03 01:06:12.042936 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:12.036446    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd"
Nov 03 01:06:12.138931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:12.131452    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-metrics-certs\") pod \"8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed\" (UID: \"8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed\") "
Nov 03 01:06:12.138931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:12.131541    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-maas-api-tls\") pod \"8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed\" (UID: \"8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed\") "
Nov 03 01:06:12.138931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:12.131707    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-kube-api-access-fs7cl\" (UniqueName: \"kubernetes.io/projected/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-kube-api-access-fs7cl\") pod \"8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed\" (UID: \"8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed\") "
Nov 03 01:06:12.138931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:12.131922    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-openshift-service-ca\") pod \"8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed\" (UID: \"8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed\") "
Nov 03 01:06:12.138931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:12.132969    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-openshift-service-ca" pod "8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed" (UID: "8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:06:12.139316 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:12.139096    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-kube-api-access-fs7cl" pod "8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed" (UID: "8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed"). InnerVolumeSpecName "kube-api-access-fs7cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:12.159184 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:12.158631    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-maas-api-tls" pod "8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed" (UID: "8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:06:12.162929 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:12.160578    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-metrics-certs" pod "8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed" (UID: "8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:06:12.234238 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:12.234130    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs7cl\" (UniqueName: \"kubernetes.io/projected/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-kube-api-access-fs7cl\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:12.234238 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:12.234178    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:12.234238 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:12.234190    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:12.234238 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:12.234198    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:12.681007 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:12.680968    5012 generic.go:356] "Generic (PLEG): container finished" podID="8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed" containerID="842cb3f77adb8df1b2e0fdd10d1467a10e247f7a985e43fb2c7c0f9c7ac7e77e" exitCode=0
Nov 03 01:06:12.681451 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:12.681021    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd"
Nov 03 01:06:12.681451 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:12.681058    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd" event={"ID":"8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed","Type":"ContainerDied","Data":"842cb3f77adb8df1b2e0fdd10d1467a10e247f7a985e43fb2c7c0f9c7ac7e77e"}
Nov 03 01:06:12.681451 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:12.681118    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd" event={"ID":"8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed","Type":"ContainerDied","Data":"278e2deb5cd3077f2e4eec005d60b22614d814dba695ddbcce1c420e5c0bae82"}
Nov 03 01:06:12.681451 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:12.681148    5012 scope.go:122] "RemoveContainer" containerID="842cb3f77adb8df1b2e0fdd10d1467a10e247f7a985e43fb2c7c0f9c7ac7e77e"
Nov 03 01:06:12.694524 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:12.694500    5012 scope.go:122] "RemoveContainer" containerID="842cb3f77adb8df1b2e0fdd10d1467a10e247f7a985e43fb2c7c0f9c7ac7e77e"
Nov 03 01:06:12.694810 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:06:12.694789    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"842cb3f77adb8df1b2e0fdd10d1467a10e247f7a985e43fb2c7c0f9c7ac7e77e\": container with ID starting with 842cb3f77adb8df1b2e0fdd10d1467a10e247f7a985e43fb2c7c0f9c7ac7e77e not found: ID does not exist" containerID="842cb3f77adb8df1b2e0fdd10d1467a10e247f7a985e43fb2c7c0f9c7ac7e77e"
Nov 03 01:06:12.694877 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:12.694825    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"842cb3f77adb8df1b2e0fdd10d1467a10e247f7a985e43fb2c7c0f9c7ac7e77e"} err="failed to get container status \"842cb3f77adb8df1b2e0fdd10d1467a10e247f7a985e43fb2c7c0f9c7ac7e77e\": rpc error: code = NotFound desc = could not find container \"842cb3f77adb8df1b2e0fdd10d1467a10e247f7a985e43fb2c7c0f9c7ac7e77e\": container with ID starting with 842cb3f77adb8df1b2e0fdd10d1467a10e247f7a985e43fb2c7c0f9c7ac7e77e not found: ID does not exist"
Nov 03 01:06:12.730770 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:12.730737    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd"]
Nov 03 01:06:12.741090 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:12.741058    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7f20f7-c8978c777-fdthd"]
Nov 03 01:06:13.248361 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:13.248323    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed" path="/var/lib/kubelet/pods/8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed/volumes"
Nov 03 01:06:21.193979 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:21.193881    5012 scope.go:122] "RemoveContainer" containerID="66145bb3ea18ba9db4a1dc1730ffb3740b59ec1d2a596b39442c6a9d9d2d3662"
Nov 03 01:06:37.386211 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:06:37.386160    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:06:38.083523 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:38.083474    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"]
Nov 03 01:06:38.083809 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:38.083770    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh" podUID="0b949944-2ed2-4a75-82d1-a2e2854e3355" containerName="istio-proxy" containerID="cri-o://73c20949c8b010fff7a490effee865357f1b0878eb699c1cf6ebe7e70dde9a1e" gracePeriod=30
Nov 03 01:06:39.874146 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:39.873679    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"]
Nov 03 01:06:39.874826 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:39.874395    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg" podUID="56c4471b-fbec-4f61-8db4-3c7f2c7298c6" containerName="istio-proxy" containerID="cri-o://c31e9e3590c361e26122cb575c09dd5747a9a3415b6627f99407e02290e15d57" gracePeriod=30
Nov 03 01:06:40.221730 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:40.221624    5012 patch_prober.go:29] interesting pod/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:06:40.221730 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:40.221692    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh" podUID="0b949944-2ed2-4a75-82d1-a2e2854e3355" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:06:41.296425 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:41.296385    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"]
Nov 03 01:06:41.296888 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:41.296585    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf" podUID="876fffce-7b9b-4aeb-8437-4c60987a5056" containerName="istio-proxy" containerID="cri-o://42799281f276d4500ea0f6601e1e08f342f3eef49f4a9c1bf004253fe30c9f09" gracePeriod=30
Nov 03 01:06:41.496607 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:41.496563    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"]
Nov 03 01:06:41.496849 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:41.496821    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6" podUID="4233c829-fa1c-443e-b7b6-9522c977fa98" containerName="istio-proxy" containerID="cri-o://cbb0eb60aea42cde08a3cea874353b19329b0c5c040fd04228daa2bc0104009e" gracePeriod=30
Nov 03 01:06:41.556687 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:41.556586    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"]
Nov 03 01:06:41.556889 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:41.556857    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm" podUID="a5e7d5fd-29e5-4719-8239-922e7a3aeeed" containerName="istio-proxy" containerID="cri-o://21e98aec444b140f4d9e751db08f11e48215768bba813015d7cf04cd3bce313e" gracePeriod=30
Nov 03 01:06:41.948491 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:41.948451    5012 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:06:41.948696 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:41.948513    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm" podUID="a5e7d5fd-29e5-4719-8239-922e7a3aeeed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:06:43.157564 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.157520    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"]
Nov 03 01:06:43.509999 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.509961    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 01:06:43.657160 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.657056    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b949944-2ed2-4a75-82d1-a2e2854e3355-kube-api-access-dg6t8\" (UniqueName: \"kubernetes.io/projected/0b949944-2ed2-4a75-82d1-a2e2854e3355-kube-api-access-dg6t8\") pod \"0b949944-2ed2-4a75-82d1-a2e2854e3355\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") "
Nov 03 01:06:43.657344 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.657277    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-credential-socket\") pod \"0b949944-2ed2-4a75-82d1-a2e2854e3355\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") "
Nov 03 01:06:43.657344 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.657316    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-envoy\") pod \"0b949944-2ed2-4a75-82d1-a2e2854e3355\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") "
Nov 03 01:06:43.657428 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.657394    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-data\") pod \"0b949944-2ed2-4a75-82d1-a2e2854e3355\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") "
Nov 03 01:06:43.657477 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.657430    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-podinfo\") pod \"0b949944-2ed2-4a75-82d1-a2e2854e3355\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") "
Nov 03 01:06:43.657576 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.657508    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-token\" (UniqueName: \"kubernetes.io/projected/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-token\") pod \"0b949944-2ed2-4a75-82d1-a2e2854e3355\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") "
Nov 03 01:06:43.657800 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.657762    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-workload-certs\") pod \"0b949944-2ed2-4a75-82d1-a2e2854e3355\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") "
Nov 03 01:06:43.657904 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.657592    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-credential-socket" pod "0b949944-2ed2-4a75-82d1-a2e2854e3355" (UID: "0b949944-2ed2-4a75-82d1-a2e2854e3355"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:43.657904 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.657807    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-workload-socket\") pod \"0b949944-2ed2-4a75-82d1-a2e2854e3355\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") "
Nov 03 01:06:43.657904 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.657693    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-data" pod "0b949944-2ed2-4a75-82d1-a2e2854e3355" (UID: "0b949944-2ed2-4a75-82d1-a2e2854e3355"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:43.657904 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.657879    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b949944-2ed2-4a75-82d1-a2e2854e3355-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b949944-2ed2-4a75-82d1-a2e2854e3355-istiod-ca-cert\") pod \"0b949944-2ed2-4a75-82d1-a2e2854e3355\" (UID: \"0b949944-2ed2-4a75-82d1-a2e2854e3355\") "
Nov 03 01:06:43.658162 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.658062    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-workload-certs" pod "0b949944-2ed2-4a75-82d1-a2e2854e3355" (UID: "0b949944-2ed2-4a75-82d1-a2e2854e3355"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:43.658249 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.658231    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-workload-socket" pod "0b949944-2ed2-4a75-82d1-a2e2854e3355" (UID: "0b949944-2ed2-4a75-82d1-a2e2854e3355"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:43.658549 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.658528    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b949944-2ed2-4a75-82d1-a2e2854e3355-istiod-ca-cert" pod "0b949944-2ed2-4a75-82d1-a2e2854e3355" (UID: "0b949944-2ed2-4a75-82d1-a2e2854e3355"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:06:43.658771 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.658756    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:43.658817 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.658774    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:43.658817 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.658786    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:43.658817 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.658815    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:43.658960 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.658825    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b949944-2ed2-4a75-82d1-a2e2854e3355-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:43.660005 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.659973    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-envoy" pod "0b949944-2ed2-4a75-82d1-a2e2854e3355" (UID: "0b949944-2ed2-4a75-82d1-a2e2854e3355"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:43.660110 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.660067    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b949944-2ed2-4a75-82d1-a2e2854e3355-kube-api-access-dg6t8" pod "0b949944-2ed2-4a75-82d1-a2e2854e3355" (UID: "0b949944-2ed2-4a75-82d1-a2e2854e3355"). InnerVolumeSpecName "kube-api-access-dg6t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:43.660164 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.660140    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-podinfo" pod "0b949944-2ed2-4a75-82d1-a2e2854e3355" (UID: "0b949944-2ed2-4a75-82d1-a2e2854e3355"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:06:43.660260 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.660242    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-token" pod "0b949944-2ed2-4a75-82d1-a2e2854e3355" (UID: "0b949944-2ed2-4a75-82d1-a2e2854e3355"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:43.760470 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.760426    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:43.760470 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.760468    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg6t8\" (UniqueName: \"kubernetes.io/projected/0b949944-2ed2-4a75-82d1-a2e2854e3355-kube-api-access-dg6t8\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:43.760711 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.760482    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:43.760711 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:43.760497    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b949944-2ed2-4a75-82d1-a2e2854e3355-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:44.079697 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:44.079587    5012 generic.go:356] "Generic (PLEG): container finished" podID="0b949944-2ed2-4a75-82d1-a2e2854e3355" containerID="73c20949c8b010fff7a490effee865357f1b0878eb699c1cf6ebe7e70dde9a1e" exitCode=0
Nov 03 01:06:44.079697 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:44.079655    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"
Nov 03 01:06:44.079697 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:44.079663    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh" event={"ID":"0b949944-2ed2-4a75-82d1-a2e2854e3355","Type":"ContainerDied","Data":"73c20949c8b010fff7a490effee865357f1b0878eb699c1cf6ebe7e70dde9a1e"}
Nov 03 01:06:44.080027 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:44.079773    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh" event={"ID":"0b949944-2ed2-4a75-82d1-a2e2854e3355","Type":"ContainerDied","Data":"fdd5474fb59464e05bf18938e34710ba4c07b4a58fadf551f2785e608c37713c"}
Nov 03 01:06:44.080027 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:44.079805    5012 scope.go:122] "RemoveContainer" containerID="73c20949c8b010fff7a490effee865357f1b0878eb699c1cf6ebe7e70dde9a1e"
Nov 03 01:06:44.093776 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:44.093750    5012 scope.go:122] "RemoveContainer" containerID="5a6d945d7210626b56ed10c44ef4956471df7943dc71b7c7ef1499cf4c2b2e7b"
Nov 03 01:06:44.106112 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:44.106086    5012 scope.go:122] "RemoveContainer" containerID="73c20949c8b010fff7a490effee865357f1b0878eb699c1cf6ebe7e70dde9a1e"
Nov 03 01:06:44.106494 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:06:44.106458    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73c20949c8b010fff7a490effee865357f1b0878eb699c1cf6ebe7e70dde9a1e\": container with ID starting with 73c20949c8b010fff7a490effee865357f1b0878eb699c1cf6ebe7e70dde9a1e not found: ID does not exist" containerID="73c20949c8b010fff7a490effee865357f1b0878eb699c1cf6ebe7e70dde9a1e"
Nov 03 01:06:44.106542 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:44.106507    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73c20949c8b010fff7a490effee865357f1b0878eb699c1cf6ebe7e70dde9a1e"} err="failed to get container status \"73c20949c8b010fff7a490effee865357f1b0878eb699c1cf6ebe7e70dde9a1e\": rpc error: code = NotFound desc = could not find container \"73c20949c8b010fff7a490effee865357f1b0878eb699c1cf6ebe7e70dde9a1e\": container with ID starting with 73c20949c8b010fff7a490effee865357f1b0878eb699c1cf6ebe7e70dde9a1e not found: ID does not exist"
Nov 03 01:06:44.106542 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:44.106526    5012 scope.go:122] "RemoveContainer" containerID="5a6d945d7210626b56ed10c44ef4956471df7943dc71b7c7ef1499cf4c2b2e7b"
Nov 03 01:06:44.106875 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:06:44.106848    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a6d945d7210626b56ed10c44ef4956471df7943dc71b7c7ef1499cf4c2b2e7b\": container with ID starting with 5a6d945d7210626b56ed10c44ef4956471df7943dc71b7c7ef1499cf4c2b2e7b not found: ID does not exist" containerID="5a6d945d7210626b56ed10c44ef4956471df7943dc71b7c7ef1499cf4c2b2e7b"
Nov 03 01:06:44.106999 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:44.106878    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a6d945d7210626b56ed10c44ef4956471df7943dc71b7c7ef1499cf4c2b2e7b"} err="failed to get container status \"5a6d945d7210626b56ed10c44ef4956471df7943dc71b7c7ef1499cf4c2b2e7b\": rpc error: code = NotFound desc = could not find container \"5a6d945d7210626b56ed10c44ef4956471df7943dc71b7c7ef1499cf4c2b2e7b\": container with ID starting with 5a6d945d7210626b56ed10c44ef4956471df7943dc71b7c7ef1499cf4c2b2e7b not found: ID does not exist"
Nov 03 01:06:44.128251 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:44.128208    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"]
Nov 03 01:06:44.138726 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:44.138680    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-9tjnh"]
Nov 03 01:06:45.096933 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.096873    5012 generic.go:356] "Generic (PLEG): container finished" podID="56c4471b-fbec-4f61-8db4-3c7f2c7298c6" containerID="c31e9e3590c361e26122cb575c09dd5747a9a3415b6627f99407e02290e15d57" exitCode=0
Nov 03 01:06:45.097496 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.097176    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg" event={"ID":"56c4471b-fbec-4f61-8db4-3c7f2c7298c6","Type":"ContainerDied","Data":"c31e9e3590c361e26122cb575c09dd5747a9a3415b6627f99407e02290e15d57"}
Nov 03 01:06:45.170587 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.170550    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 01:06:45.255007 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.254956    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b949944-2ed2-4a75-82d1-a2e2854e3355" path="/var/lib/kubelet/pods/0b949944-2ed2-4a75-82d1-a2e2854e3355/volumes"
Nov 03 01:06:45.283494 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.283458    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-credential-socket\") pod \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") "
Nov 03 01:06:45.283726 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.283562    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-kube-api-access-vxlm4\" (UniqueName: \"kubernetes.io/projected/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-kube-api-access-vxlm4\") pod \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") "
Nov 03 01:06:45.283726 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.283620    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-envoy\") pod \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") "
Nov 03 01:06:45.283726 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.283720    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-workload-socket\") pod \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") "
Nov 03 01:06:45.283927 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.283806    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-token\" (UniqueName: \"kubernetes.io/projected/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-token\") pod \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") "
Nov 03 01:06:45.283927 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.283819    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-credential-socket" pod "56c4471b-fbec-4f61-8db4-3c7f2c7298c6" (UID: "56c4471b-fbec-4f61-8db4-3c7f2c7298c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:45.284067 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.284039    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-workload-socket" pod "56c4471b-fbec-4f61-8db4-3c7f2c7298c6" (UID: "56c4471b-fbec-4f61-8db4-3c7f2c7298c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:45.284135 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.284050    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-podinfo\") pod \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") "
Nov 03 01:06:45.284135 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.284121    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istiod-ca-cert\") pod \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") "
Nov 03 01:06:45.284261 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.284157    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-workload-certs\") pod \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") "
Nov 03 01:06:45.284261 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.284186    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-data\") pod \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\" (UID: \"56c4471b-fbec-4f61-8db4-3c7f2c7298c6\") "
Nov 03 01:06:45.284578 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.284550    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istiod-ca-cert" pod "56c4471b-fbec-4f61-8db4-3c7f2c7298c6" (UID: "56c4471b-fbec-4f61-8db4-3c7f2c7298c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:06:45.284838 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.284813    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-data" pod "56c4471b-fbec-4f61-8db4-3c7f2c7298c6" (UID: "56c4471b-fbec-4f61-8db4-3c7f2c7298c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:45.284944 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.284844    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-workload-certs" pod "56c4471b-fbec-4f61-8db4-3c7f2c7298c6" (UID: "56c4471b-fbec-4f61-8db4-3c7f2c7298c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:45.285250 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.285225    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:45.285345 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.285255    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:45.285345 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.285270    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:45.285345 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.285285    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:45.285345 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.285299    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:45.286167 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.286146    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-kube-api-access-vxlm4" pod "56c4471b-fbec-4f61-8db4-3c7f2c7298c6" (UID: "56c4471b-fbec-4f61-8db4-3c7f2c7298c6"). InnerVolumeSpecName "kube-api-access-vxlm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:45.286275 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.286150    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-envoy" pod "56c4471b-fbec-4f61-8db4-3c7f2c7298c6" (UID: "56c4471b-fbec-4f61-8db4-3c7f2c7298c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:45.286326 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.286280    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-token" pod "56c4471b-fbec-4f61-8db4-3c7f2c7298c6" (UID: "56c4471b-fbec-4f61-8db4-3c7f2c7298c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:45.286326 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.286300    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-podinfo" pod "56c4471b-fbec-4f61-8db4-3c7f2c7298c6" (UID: "56c4471b-fbec-4f61-8db4-3c7f2c7298c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:06:45.387586 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.387390    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxlm4\" (UniqueName: \"kubernetes.io/projected/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-kube-api-access-vxlm4\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:45.387586 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.387431    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:45.387586 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.387446    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:45.387586 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:45.387458    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c4471b-fbec-4f61-8db4-3c7f2c7298c6-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:46.116604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.116562    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"
Nov 03 01:06:46.116604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.116586    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg" event={"ID":"56c4471b-fbec-4f61-8db4-3c7f2c7298c6","Type":"ContainerDied","Data":"2d140c7fc5eb5d6ff1f3a0e6c32e1f0b81c0f9b1f1621b730b347b457bbfd62c"}
Nov 03 01:06:46.117275 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.116681    5012 scope.go:122] "RemoveContainer" containerID="c31e9e3590c361e26122cb575c09dd5747a9a3415b6627f99407e02290e15d57"
Nov 03 01:06:46.179136 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.179086    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"]
Nov 03 01:06:46.192701 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.192654    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-7sjhg"]
Nov 03 01:06:46.727949 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.727480    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 01:06:46.831897 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.831854    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-envoy\") pod \"876fffce-7b9b-4aeb-8437-4c60987a5056\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") "
Nov 03 01:06:46.832125 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.831956    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-credential-socket\") pod \"876fffce-7b9b-4aeb-8437-4c60987a5056\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") "
Nov 03 01:06:46.832125 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.831995    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/876fffce-7b9b-4aeb-8437-4c60987a5056-kube-api-access-q6cg9\" (UniqueName: \"kubernetes.io/projected/876fffce-7b9b-4aeb-8437-4c60987a5056-kube-api-access-q6cg9\") pod \"876fffce-7b9b-4aeb-8437-4c60987a5056\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") "
Nov 03 01:06:46.832125 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.832101    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/876fffce-7b9b-4aeb-8437-4c60987a5056-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/876fffce-7b9b-4aeb-8437-4c60987a5056-istiod-ca-cert\") pod \"876fffce-7b9b-4aeb-8437-4c60987a5056\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") "
Nov 03 01:06:46.832291 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.832247    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-workload-certs\") pod \"876fffce-7b9b-4aeb-8437-4c60987a5056\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") "
Nov 03 01:06:46.832350 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.832301    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-workload-socket\") pod \"876fffce-7b9b-4aeb-8437-4c60987a5056\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") "
Nov 03 01:06:46.832411 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.832345    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-podinfo\") pod \"876fffce-7b9b-4aeb-8437-4c60987a5056\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") "
Nov 03 01:06:46.832411 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.832387    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-token\" (UniqueName: \"kubernetes.io/projected/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-token\") pod \"876fffce-7b9b-4aeb-8437-4c60987a5056\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") "
Nov 03 01:06:46.832512 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.832425    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-data\") pod \"876fffce-7b9b-4aeb-8437-4c60987a5056\" (UID: \"876fffce-7b9b-4aeb-8437-4c60987a5056\") "
Nov 03 01:06:46.832570 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.832511    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-credential-socket" pod "876fffce-7b9b-4aeb-8437-4c60987a5056" (UID: "876fffce-7b9b-4aeb-8437-4c60987a5056"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:46.833152 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.833123    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:46.833659 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.833280    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-workload-certs" pod "876fffce-7b9b-4aeb-8437-4c60987a5056" (UID: "876fffce-7b9b-4aeb-8437-4c60987a5056"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:46.833659 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.833366    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-data" pod "876fffce-7b9b-4aeb-8437-4c60987a5056" (UID: "876fffce-7b9b-4aeb-8437-4c60987a5056"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:46.833659 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.833578    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-workload-socket" pod "876fffce-7b9b-4aeb-8437-4c60987a5056" (UID: "876fffce-7b9b-4aeb-8437-4c60987a5056"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:46.833659 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.833648    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/876fffce-7b9b-4aeb-8437-4c60987a5056-istiod-ca-cert" pod "876fffce-7b9b-4aeb-8437-4c60987a5056" (UID: "876fffce-7b9b-4aeb-8437-4c60987a5056"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:06:46.838325 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.838286    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/876fffce-7b9b-4aeb-8437-4c60987a5056-kube-api-access-q6cg9" pod "876fffce-7b9b-4aeb-8437-4c60987a5056" (UID: "876fffce-7b9b-4aeb-8437-4c60987a5056"). InnerVolumeSpecName "kube-api-access-q6cg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:46.840859 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.840702    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-envoy" pod "876fffce-7b9b-4aeb-8437-4c60987a5056" (UID: "876fffce-7b9b-4aeb-8437-4c60987a5056"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:46.840859 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.840771    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-token" pod "876fffce-7b9b-4aeb-8437-4c60987a5056" (UID: "876fffce-7b9b-4aeb-8437-4c60987a5056"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:46.841848 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.841807    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-podinfo" pod "876fffce-7b9b-4aeb-8437-4c60987a5056" (UID: "876fffce-7b9b-4aeb-8437-4c60987a5056"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:06:46.938327 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.938277    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/876fffce-7b9b-4aeb-8437-4c60987a5056-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:46.938598 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.938577    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:46.938747 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.938730    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:46.938860 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.938845    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:46.939104 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.939084    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:46.939223 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.939210    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:46.939652 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.939632    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/876fffce-7b9b-4aeb-8437-4c60987a5056-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:46.939778 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.939761    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6cg9\" (UniqueName: \"kubernetes.io/projected/876fffce-7b9b-4aeb-8437-4c60987a5056-kube-api-access-q6cg9\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:46.939879 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:46.939597    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 01:06:47.047481 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.047376    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-data\") pod \"4233c829-fa1c-443e-b7b6-9522c977fa98\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") "
Nov 03 01:06:47.047816 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.047791    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-workload-certs\") pod \"4233c829-fa1c-443e-b7b6-9522c977fa98\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") "
Nov 03 01:06:47.048018 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.047993    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-podinfo\") pod \"4233c829-fa1c-443e-b7b6-9522c977fa98\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") "
Nov 03 01:06:47.048185 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.048156    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-workload-certs" pod "4233c829-fa1c-443e-b7b6-9522c977fa98" (UID: "4233c829-fa1c-443e-b7b6-9522c977fa98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:47.048261 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.047902    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-data" pod "4233c829-fa1c-443e-b7b6-9522c977fa98" (UID: "4233c829-fa1c-443e-b7b6-9522c977fa98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:47.048381 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.048360    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-envoy\") pod \"4233c829-fa1c-443e-b7b6-9522c977fa98\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") "
Nov 03 01:06:47.048526 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.048507    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-workload-socket\") pod \"4233c829-fa1c-443e-b7b6-9522c977fa98\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") "
Nov 03 01:06:47.048686 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.048663    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4233c829-fa1c-443e-b7b6-9522c977fa98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4233c829-fa1c-443e-b7b6-9522c977fa98-istiod-ca-cert\") pod \"4233c829-fa1c-443e-b7b6-9522c977fa98\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") "
Nov 03 01:06:47.048877 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.048860    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-token\" (UniqueName: \"kubernetes.io/projected/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-token\") pod \"4233c829-fa1c-443e-b7b6-9522c977fa98\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") "
Nov 03 01:06:47.049131 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.049109    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4233c829-fa1c-443e-b7b6-9522c977fa98-kube-api-access-4t4gm\" (UniqueName: \"kubernetes.io/projected/4233c829-fa1c-443e-b7b6-9522c977fa98-kube-api-access-4t4gm\") pod \"4233c829-fa1c-443e-b7b6-9522c977fa98\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") "
Nov 03 01:06:47.049289 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.049271    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-credential-socket\") pod \"4233c829-fa1c-443e-b7b6-9522c977fa98\" (UID: \"4233c829-fa1c-443e-b7b6-9522c977fa98\") "
Nov 03 01:06:47.050318 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.050292    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.050486 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.050460    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.050594 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.050395    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4233c829-fa1c-443e-b7b6-9522c977fa98-istiod-ca-cert" pod "4233c829-fa1c-443e-b7b6-9522c977fa98" (UID: "4233c829-fa1c-443e-b7b6-9522c977fa98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:06:47.050738 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.050711    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-workload-socket" pod "4233c829-fa1c-443e-b7b6-9522c977fa98" (UID: "4233c829-fa1c-443e-b7b6-9522c977fa98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:47.051092 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.051058    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-credential-socket" pod "4233c829-fa1c-443e-b7b6-9522c977fa98" (UID: "4233c829-fa1c-443e-b7b6-9522c977fa98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:47.057954 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.057179    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-token" pod "4233c829-fa1c-443e-b7b6-9522c977fa98" (UID: "4233c829-fa1c-443e-b7b6-9522c977fa98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:47.073938 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.070486    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4233c829-fa1c-443e-b7b6-9522c977fa98-kube-api-access-4t4gm" pod "4233c829-fa1c-443e-b7b6-9522c977fa98" (UID: "4233c829-fa1c-443e-b7b6-9522c977fa98"). InnerVolumeSpecName "kube-api-access-4t4gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:47.075349 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.075288    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-podinfo" pod "4233c829-fa1c-443e-b7b6-9522c977fa98" (UID: "4233c829-fa1c-443e-b7b6-9522c977fa98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:06:47.081895 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.081661    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-envoy" pod "4233c829-fa1c-443e-b7b6-9522c977fa98" (UID: "4233c829-fa1c-443e-b7b6-9522c977fa98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:47.184068 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.184026    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.184692 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.184665    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.184819 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.184803    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.184960 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.184939    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4233c829-fa1c-443e-b7b6-9522c977fa98-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.185081 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.185068    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4233c829-fa1c-443e-b7b6-9522c977fa98-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.185193 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.185179    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t4gm\" (UniqueName: \"kubernetes.io/projected/4233c829-fa1c-443e-b7b6-9522c977fa98-kube-api-access-4t4gm\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.185288 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.185274    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4233c829-fa1c-443e-b7b6-9522c977fa98-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.209935 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.209275    5012 generic.go:356] "Generic (PLEG): container finished" podID="876fffce-7b9b-4aeb-8437-4c60987a5056" containerID="42799281f276d4500ea0f6601e1e08f342f3eef49f4a9c1bf004253fe30c9f09" exitCode=0
Nov 03 01:06:47.209935 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.209405    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf" event={"ID":"876fffce-7b9b-4aeb-8437-4c60987a5056","Type":"ContainerDied","Data":"42799281f276d4500ea0f6601e1e08f342f3eef49f4a9c1bf004253fe30c9f09"}
Nov 03 01:06:47.209935 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.209459    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf" event={"ID":"876fffce-7b9b-4aeb-8437-4c60987a5056","Type":"ContainerDied","Data":"35dd00a1792753bdb31c52c5c9c32c89e1ef10eedb20d92e6755cbe041913401"}
Nov 03 01:06:47.209935 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.209639    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"
Nov 03 01:06:47.222936 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.209492    5012 scope.go:122] "RemoveContainer" containerID="42799281f276d4500ea0f6601e1e08f342f3eef49f4a9c1bf004253fe30c9f09"
Nov 03 01:06:47.257937 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.257178    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 01:06:47.267934 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.258355    5012 generic.go:356] "Generic (PLEG): container finished" podID="4233c829-fa1c-443e-b7b6-9522c977fa98" containerID="cbb0eb60aea42cde08a3cea874353b19329b0c5c040fd04228daa2bc0104009e" exitCode=0
Nov 03 01:06:47.267934 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.258578    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6" event={"ID":"4233c829-fa1c-443e-b7b6-9522c977fa98","Type":"ContainerDied","Data":"cbb0eb60aea42cde08a3cea874353b19329b0c5c040fd04228daa2bc0104009e"}
Nov 03 01:06:47.267934 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.258635    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6" event={"ID":"4233c829-fa1c-443e-b7b6-9522c977fa98","Type":"ContainerDied","Data":"0d1b5507c2d019218bb65979167d26440a4fec74cc0cce932137284d2757c307"}
Nov 03 01:06:47.267934 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.258729    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"
Nov 03 01:06:47.297997 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.296380    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-token\" (UniqueName: \"kubernetes.io/projected/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-token\") pod \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") "
Nov 03 01:06:47.297997 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.296477    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-envoy\") pod \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") "
Nov 03 01:06:47.297997 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.296579    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-kube-api-access-bn8k2\" (UniqueName: \"kubernetes.io/projected/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-kube-api-access-bn8k2\") pod \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") "
Nov 03 01:06:47.297997 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.296628    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-credential-socket\") pod \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") "
Nov 03 01:06:47.297997 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.296777    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istiod-ca-cert\") pod \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") "
Nov 03 01:06:47.297997 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.296806    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-data\") pod \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") "
Nov 03 01:06:47.297997 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.296836    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-podinfo\") pod \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") "
Nov 03 01:06:47.297997 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.296894    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-workload-certs\") pod \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") "
Nov 03 01:06:47.298715 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.298135    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-data" pod "a5e7d5fd-29e5-4719-8239-922e7a3aeeed" (UID: "a5e7d5fd-29e5-4719-8239-922e7a3aeeed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:47.298715 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.298501    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istiod-ca-cert" pod "a5e7d5fd-29e5-4719-8239-922e7a3aeeed" (UID: "a5e7d5fd-29e5-4719-8239-922e7a3aeeed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:06:47.299222 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.299192    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-workload-socket\") pod \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\" (UID: \"a5e7d5fd-29e5-4719-8239-922e7a3aeeed\") "
Nov 03 01:06:47.300226 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.300198    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-credential-socket" pod "a5e7d5fd-29e5-4719-8239-922e7a3aeeed" (UID: "a5e7d5fd-29e5-4719-8239-922e7a3aeeed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:47.300729 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.300705    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-workload-socket" pod "a5e7d5fd-29e5-4719-8239-922e7a3aeeed" (UID: "a5e7d5fd-29e5-4719-8239-922e7a3aeeed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:47.301081 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.301060    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-workload-certs" pod "a5e7d5fd-29e5-4719-8239-922e7a3aeeed" (UID: "a5e7d5fd-29e5-4719-8239-922e7a3aeeed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:47.302261 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.302235    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.302414 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.302396    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.302542 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.302525    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.302649 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.302636    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.302767 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.302753    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.328852 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.328808    5012 generic.go:356] "Generic (PLEG): container finished" podID="a5e7d5fd-29e5-4719-8239-922e7a3aeeed" containerID="21e98aec444b140f4d9e751db08f11e48215768bba813015d7cf04cd3bce313e" exitCode=0
Nov 03 01:06:47.423272 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.380230    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"
Nov 03 01:06:47.547291 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.537535    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-kube-api-access-bn8k2" pod "a5e7d5fd-29e5-4719-8239-922e7a3aeeed" (UID: "a5e7d5fd-29e5-4719-8239-922e7a3aeeed"). InnerVolumeSpecName "kube-api-access-bn8k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:47.571082 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.565322    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-podinfo" pod "a5e7d5fd-29e5-4719-8239-922e7a3aeeed" (UID: "a5e7d5fd-29e5-4719-8239-922e7a3aeeed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:06:47.578186 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.578137    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-token" pod "a5e7d5fd-29e5-4719-8239-922e7a3aeeed" (UID: "a5e7d5fd-29e5-4719-8239-922e7a3aeeed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:47.673957 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.666370    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-envoy" pod "a5e7d5fd-29e5-4719-8239-922e7a3aeeed" (UID: "a5e7d5fd-29e5-4719-8239-922e7a3aeeed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:47.680855 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.680813    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56c4471b-fbec-4f61-8db4-3c7f2c7298c6" path="/var/lib/kubelet/pods/56c4471b-fbec-4f61-8db4-3c7f2c7298c6/volumes"
Nov 03 01:06:47.681680 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.681643    5012 scope.go:122] "RemoveContainer" containerID="42799281f276d4500ea0f6601e1e08f342f3eef49f4a9c1bf004253fe30c9f09"
Nov 03 01:06:47.724938 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.702042    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm" event={"ID":"a5e7d5fd-29e5-4719-8239-922e7a3aeeed","Type":"ContainerDied","Data":"21e98aec444b140f4d9e751db08f11e48215768bba813015d7cf04cd3bce313e"}
Nov 03 01:06:47.724938 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.702154    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"]
Nov 03 01:06:47.724938 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.702211    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm" event={"ID":"a5e7d5fd-29e5-4719-8239-922e7a3aeeed","Type":"ContainerDied","Data":"b6563a31625bbd1a193e2a8d9d35602969e5648bcaa2d488ca9e1bee1fa42bb6"}
Nov 03 01:06:47.724938 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.702484    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv" podUID="d79c28c1-9ce7-4040-90a7-6a423b1e349e" containerName="istio-proxy" containerID="cri-o://cf7739acf87ac442a550d08932dc2316973b5654f4fefb29d969764289ab5eb0" gracePeriod=30
Nov 03 01:06:47.724938 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:06:47.703352    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42799281f276d4500ea0f6601e1e08f342f3eef49f4a9c1bf004253fe30c9f09\": container with ID starting with 42799281f276d4500ea0f6601e1e08f342f3eef49f4a9c1bf004253fe30c9f09 not found: ID does not exist" containerID="42799281f276d4500ea0f6601e1e08f342f3eef49f4a9c1bf004253fe30c9f09"
Nov 03 01:06:47.724938 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.703394    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42799281f276d4500ea0f6601e1e08f342f3eef49f4a9c1bf004253fe30c9f09"} err="failed to get container status \"42799281f276d4500ea0f6601e1e08f342f3eef49f4a9c1bf004253fe30c9f09\": rpc error: code = NotFound desc = could not find container \"42799281f276d4500ea0f6601e1e08f342f3eef49f4a9c1bf004253fe30c9f09\": container with ID starting with 42799281f276d4500ea0f6601e1e08f342f3eef49f4a9c1bf004253fe30c9f09 not found: ID does not exist"
Nov 03 01:06:47.724938 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.703422    5012 scope.go:122] "RemoveContainer" containerID="cbb0eb60aea42cde08a3cea874353b19329b0c5c040fd04228daa2bc0104009e"
Nov 03 01:06:47.766537 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.759139    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"]
Nov 03 01:06:47.766537 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.764752    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.766537 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.764782    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.766537 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.764798    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.766537 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.764812    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn8k2\" (UniqueName: \"kubernetes.io/projected/a5e7d5fd-29e5-4719-8239-922e7a3aeeed-kube-api-access-bn8k2\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.779228 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.770509    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x9mrf"]
Nov 03 01:06:47.885642 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.885252    5012 scope.go:122] "RemoveContainer" containerID="b41b4b89c96bab9fe680c62048ca2a8bf0d67e17835e735f5ac578d87d9a287f"
Nov 03 01:06:47.984620 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.984581    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"]
Nov 03 01:06:47.985786 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:47.985763    5012 scope.go:122] "RemoveContainer" containerID="cbb0eb60aea42cde08a3cea874353b19329b0c5c040fd04228daa2bc0104009e"
Nov 03 01:06:48.010437 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:06:47.991445    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb0eb60aea42cde08a3cea874353b19329b0c5c040fd04228daa2bc0104009e\": container with ID starting with cbb0eb60aea42cde08a3cea874353b19329b0c5c040fd04228daa2bc0104009e not found: ID does not exist" containerID="cbb0eb60aea42cde08a3cea874353b19329b0c5c040fd04228daa2bc0104009e"
Nov 03 01:06:48.010825 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:48.010773    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb0eb60aea42cde08a3cea874353b19329b0c5c040fd04228daa2bc0104009e"} err="failed to get container status \"cbb0eb60aea42cde08a3cea874353b19329b0c5c040fd04228daa2bc0104009e\": rpc error: code = NotFound desc = could not find container \"cbb0eb60aea42cde08a3cea874353b19329b0c5c040fd04228daa2bc0104009e\": container with ID starting with cbb0eb60aea42cde08a3cea874353b19329b0c5c040fd04228daa2bc0104009e not found: ID does not exist"
Nov 03 01:06:48.026020 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:48.025970    5012 scope.go:122] "RemoveContainer" containerID="b41b4b89c96bab9fe680c62048ca2a8bf0d67e17835e735f5ac578d87d9a287f"
Nov 03 01:06:48.026424 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:48.006380    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sjthm"]
Nov 03 01:06:48.035988 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:06:48.027353    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b41b4b89c96bab9fe680c62048ca2a8bf0d67e17835e735f5ac578d87d9a287f\": container with ID starting with b41b4b89c96bab9fe680c62048ca2a8bf0d67e17835e735f5ac578d87d9a287f not found: ID does not exist" containerID="b41b4b89c96bab9fe680c62048ca2a8bf0d67e17835e735f5ac578d87d9a287f"
Nov 03 01:06:48.035988 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:48.027399    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b41b4b89c96bab9fe680c62048ca2a8bf0d67e17835e735f5ac578d87d9a287f"} err="failed to get container status \"b41b4b89c96bab9fe680c62048ca2a8bf0d67e17835e735f5ac578d87d9a287f\": rpc error: code = NotFound desc = could not find container \"b41b4b89c96bab9fe680c62048ca2a8bf0d67e17835e735f5ac578d87d9a287f\": container with ID starting with b41b4b89c96bab9fe680c62048ca2a8bf0d67e17835e735f5ac578d87d9a287f not found: ID does not exist"
Nov 03 01:06:48.035988 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:48.027428    5012 scope.go:122] "RemoveContainer" containerID="21e98aec444b140f4d9e751db08f11e48215768bba813015d7cf04cd3bce313e"
Nov 03 01:06:48.077977 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:48.077936    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"]
Nov 03 01:06:48.081174 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:48.081105    5012 scope.go:122] "RemoveContainer" containerID="21e98aec444b140f4d9e751db08f11e48215768bba813015d7cf04cd3bce313e"
Nov 03 01:06:48.081502 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:06:48.081475    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e98aec444b140f4d9e751db08f11e48215768bba813015d7cf04cd3bce313e\": container with ID starting with 21e98aec444b140f4d9e751db08f11e48215768bba813015d7cf04cd3bce313e not found: ID does not exist" containerID="21e98aec444b140f4d9e751db08f11e48215768bba813015d7cf04cd3bce313e"
Nov 03 01:06:48.081561 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:48.081513    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e98aec444b140f4d9e751db08f11e48215768bba813015d7cf04cd3bce313e"} err="failed to get container status \"21e98aec444b140f4d9e751db08f11e48215768bba813015d7cf04cd3bce313e\": rpc error: code = NotFound desc = could not find container \"21e98aec444b140f4d9e751db08f11e48215768bba813015d7cf04cd3bce313e\": container with ID starting with 21e98aec444b140f4d9e751db08f11e48215768bba813015d7cf04cd3bce313e not found: ID does not exist"
Nov 03 01:06:48.083516 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:48.083491    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-q6rc6"]
Nov 03 01:06:49.246541 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:49.246498    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4233c829-fa1c-443e-b7b6-9522c977fa98" path="/var/lib/kubelet/pods/4233c829-fa1c-443e-b7b6-9522c977fa98/volumes"
Nov 03 01:06:49.247274 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:49.247249    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="876fffce-7b9b-4aeb-8437-4c60987a5056" path="/var/lib/kubelet/pods/876fffce-7b9b-4aeb-8437-4c60987a5056/volumes"
Nov 03 01:06:49.247821 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:49.247803    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5e7d5fd-29e5-4719-8239-922e7a3aeeed" path="/var/lib/kubelet/pods/a5e7d5fd-29e5-4719-8239-922e7a3aeeed/volumes"
Nov 03 01:06:52.997842 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:52.997806    5012 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body=
Nov 03 01:06:52.998284 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:52.997865    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv" podUID="d79c28c1-9ce7-4040-90a7-6a423b1e349e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused"
Nov 03 01:06:53.447517 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.447488    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 01:06:53.515222 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.515188    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d79c28c1-9ce7-4040-90a7-6a423b1e349e-kube-api-access-wn2pl\" (UniqueName: \"kubernetes.io/projected/d79c28c1-9ce7-4040-90a7-6a423b1e349e-kube-api-access-wn2pl\") pod \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") "
Nov 03 01:06:53.515412 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.515241    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-workload-certs\") pod \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") "
Nov 03 01:06:53.515412 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.515284    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-workload-socket\") pod \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") "
Nov 03 01:06:53.515412 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.515367    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istiod-ca-cert\") pod \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") "
Nov 03 01:06:53.515586 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.515492    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-envoy\") pod \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") "
Nov 03 01:06:53.515586 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.515531    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-data\") pod \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") "
Nov 03 01:06:53.515586 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.515570    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-credential-socket\") pod \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") "
Nov 03 01:06:53.515733 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.515587    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-workload-certs" pod "d79c28c1-9ce7-4040-90a7-6a423b1e349e" (UID: "d79c28c1-9ce7-4040-90a7-6a423b1e349e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:53.515733 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.515606    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-token\" (UniqueName: \"kubernetes.io/projected/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-token\") pod \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") "
Nov 03 01:06:53.515846 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.515738    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-podinfo\") pod \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\" (UID: \"d79c28c1-9ce7-4040-90a7-6a423b1e349e\") "
Nov 03 01:06:53.515846 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.515750    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istiod-ca-cert" pod "d79c28c1-9ce7-4040-90a7-6a423b1e349e" (UID: "d79c28c1-9ce7-4040-90a7-6a423b1e349e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:06:53.515846 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.515803    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-data" pod "d79c28c1-9ce7-4040-90a7-6a423b1e349e" (UID: "d79c28c1-9ce7-4040-90a7-6a423b1e349e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:53.516030 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.515888    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-credential-socket" pod "d79c28c1-9ce7-4040-90a7-6a423b1e349e" (UID: "d79c28c1-9ce7-4040-90a7-6a423b1e349e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:53.516030 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.515963    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-workload-socket" pod "d79c28c1-9ce7-4040-90a7-6a423b1e349e" (UID: "d79c28c1-9ce7-4040-90a7-6a423b1e349e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:53.516356 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.516325    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:53.516356 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.516351    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:53.516543 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.516366    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:53.516543 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.516379    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:53.516543 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.516391    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:53.518396 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.518368    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d79c28c1-9ce7-4040-90a7-6a423b1e349e-kube-api-access-wn2pl" pod "d79c28c1-9ce7-4040-90a7-6a423b1e349e" (UID: "d79c28c1-9ce7-4040-90a7-6a423b1e349e"). InnerVolumeSpecName "kube-api-access-wn2pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:53.518508 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.518399    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-token" pod "d79c28c1-9ce7-4040-90a7-6a423b1e349e" (UID: "d79c28c1-9ce7-4040-90a7-6a423b1e349e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:53.518508 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.518464    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-podinfo" pod "d79c28c1-9ce7-4040-90a7-6a423b1e349e" (UID: "d79c28c1-9ce7-4040-90a7-6a423b1e349e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:06:53.518508 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.518494    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-envoy" pod "d79c28c1-9ce7-4040-90a7-6a423b1e349e" (UID: "d79c28c1-9ce7-4040-90a7-6a423b1e349e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:53.617852 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.617814    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:53.617852 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.617848    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:53.618090 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.617862    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d79c28c1-9ce7-4040-90a7-6a423b1e349e-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:53.618090 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.617875    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn2pl\" (UniqueName: \"kubernetes.io/projected/d79c28c1-9ce7-4040-90a7-6a423b1e349e-kube-api-access-wn2pl\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:53.786312 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.786237    5012 generic.go:356] "Generic (PLEG): container finished" podID="d79c28c1-9ce7-4040-90a7-6a423b1e349e" containerID="cf7739acf87ac442a550d08932dc2316973b5654f4fefb29d969764289ab5eb0" exitCode=0
Nov 03 01:06:53.786312 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.786285    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"
Nov 03 01:06:53.786500 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.786308    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv" event={"ID":"d79c28c1-9ce7-4040-90a7-6a423b1e349e","Type":"ContainerDied","Data":"cf7739acf87ac442a550d08932dc2316973b5654f4fefb29d969764289ab5eb0"}
Nov 03 01:06:53.786500 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.786356    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv" event={"ID":"d79c28c1-9ce7-4040-90a7-6a423b1e349e","Type":"ContainerDied","Data":"59531835c357780aece34e5bb5cf7f4a9d58149a8a62506fa890e3a617540d73"}
Nov 03 01:06:53.786500 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.786377    5012 scope.go:122] "RemoveContainer" containerID="cf7739acf87ac442a550d08932dc2316973b5654f4fefb29d969764289ab5eb0"
Nov 03 01:06:53.798468 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.798447    5012 scope.go:122] "RemoveContainer" containerID="cf7739acf87ac442a550d08932dc2316973b5654f4fefb29d969764289ab5eb0"
Nov 03 01:06:53.798777 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:06:53.798753    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf7739acf87ac442a550d08932dc2316973b5654f4fefb29d969764289ab5eb0\": container with ID starting with cf7739acf87ac442a550d08932dc2316973b5654f4fefb29d969764289ab5eb0 not found: ID does not exist" containerID="cf7739acf87ac442a550d08932dc2316973b5654f4fefb29d969764289ab5eb0"
Nov 03 01:06:53.798852 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.798789    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf7739acf87ac442a550d08932dc2316973b5654f4fefb29d969764289ab5eb0"} err="failed to get container status \"cf7739acf87ac442a550d08932dc2316973b5654f4fefb29d969764289ab5eb0\": rpc error: code = NotFound desc = could not find container \"cf7739acf87ac442a550d08932dc2316973b5654f4fefb29d969764289ab5eb0\": container with ID starting with cf7739acf87ac442a550d08932dc2316973b5654f4fefb29d969764289ab5eb0 not found: ID does not exist"
Nov 03 01:06:53.835195 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.835158    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"]
Nov 03 01:06:53.845426 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:53.845404    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-s5msv"]
Nov 03 01:06:55.246900 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:55.246843    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d79c28c1-9ce7-4040-90a7-6a423b1e349e" path="/var/lib/kubelet/pods/d79c28c1-9ce7-4040-90a7-6a423b1e349e/volumes"
Nov 03 01:06:58.180736 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.180687    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57" podUID="b8cf1798-5790-4367-aa58-d8bc7ecc0932" containerName="main" containerID="cri-o://0d8baa32d49b8ef323c3466c7f4543717c027a351c17d9d0eff2d592ab607fd3" gracePeriod=45
Nov 03 01:06:58.467870 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.467844    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:06:58.589942 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.589881    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8cf1798-5790-4367-aa58-d8bc7ecc0932-kube-api-access-hwqxv\" (UniqueName: \"kubernetes.io/projected/b8cf1798-5790-4367-aa58-d8bc7ecc0932-kube-api-access-hwqxv\") pod \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\" (UID: \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\") "
Nov 03 01:06:58.590123 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.589983    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8cf1798-5790-4367-aa58-d8bc7ecc0932-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8cf1798-5790-4367-aa58-d8bc7ecc0932-tls-certs\") pod \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\" (UID: \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\") "
Nov 03 01:06:58.590196 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.590137    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-tmp-dir\") pod \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\" (UID: \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\") "
Nov 03 01:06:58.590196 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.590174    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-dshm\") pod \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\" (UID: \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\") "
Nov 03 01:06:58.590337 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.590322    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-model-cache\") pod \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\" (UID: \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\") "
Nov 03 01:06:58.590397 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.590374    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-tmp-dir" pod "b8cf1798-5790-4367-aa58-d8bc7ecc0932" (UID: "b8cf1798-5790-4367-aa58-d8bc7ecc0932"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:58.590458 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.590414    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-home\" (UniqueName: \"kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-home\") pod \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\" (UID: \"b8cf1798-5790-4367-aa58-d8bc7ecc0932\") "
Nov 03 01:06:58.590674 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.590647    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-model-cache" pod "b8cf1798-5790-4367-aa58-d8bc7ecc0932" (UID: "b8cf1798-5790-4367-aa58-d8bc7ecc0932"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:58.590743 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.590689    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-home" pod "b8cf1798-5790-4367-aa58-d8bc7ecc0932" (UID: "b8cf1798-5790-4367-aa58-d8bc7ecc0932"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:58.591030 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.591008    5012 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-tmp-dir\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:58.591079 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.591037    5012 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-model-cache\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:58.591079 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.591051    5012 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-home\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:58.592188 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.592167    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8cf1798-5790-4367-aa58-d8bc7ecc0932-kube-api-access-hwqxv" pod "b8cf1798-5790-4367-aa58-d8bc7ecc0932" (UID: "b8cf1798-5790-4367-aa58-d8bc7ecc0932"). InnerVolumeSpecName "kube-api-access-hwqxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:58.592188 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.592177    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-dshm" pod "b8cf1798-5790-4367-aa58-d8bc7ecc0932" (UID: "b8cf1798-5790-4367-aa58-d8bc7ecc0932"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:58.592296 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.592276    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8cf1798-5790-4367-aa58-d8bc7ecc0932-tls-certs" pod "b8cf1798-5790-4367-aa58-d8bc7ecc0932" (UID: "b8cf1798-5790-4367-aa58-d8bc7ecc0932"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:06:58.692455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.692394    5012 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8cf1798-5790-4367-aa58-d8bc7ecc0932-dshm\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:58.692455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.692420    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwqxv\" (UniqueName: \"kubernetes.io/projected/b8cf1798-5790-4367-aa58-d8bc7ecc0932-kube-api-access-hwqxv\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:58.692455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.692430    5012 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8cf1798-5790-4367-aa58-d8bc7ecc0932-tls-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:06:58.836139 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.836109    5012 generic.go:356] "Generic (PLEG): container finished" podID="b8cf1798-5790-4367-aa58-d8bc7ecc0932" containerID="0d8baa32d49b8ef323c3466c7f4543717c027a351c17d9d0eff2d592ab607fd3" exitCode=0
Nov 03 01:06:58.836279 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.836161    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"
Nov 03 01:06:58.836279 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.836190    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57" event={"ID":"b8cf1798-5790-4367-aa58-d8bc7ecc0932","Type":"ContainerDied","Data":"0d8baa32d49b8ef323c3466c7f4543717c027a351c17d9d0eff2d592ab607fd3"}
Nov 03 01:06:58.836279 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.836236    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57" event={"ID":"b8cf1798-5790-4367-aa58-d8bc7ecc0932","Type":"ContainerDied","Data":"7d02c08d0ed63d1fd2d57c9bb0a098de9e5d9b7edccccca0a143e64463e8db6d"}
Nov 03 01:06:58.836279 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.836257    5012 scope.go:122] "RemoveContainer" containerID="0d8baa32d49b8ef323c3466c7f4543717c027a351c17d9d0eff2d592ab607fd3"
Nov 03 01:06:58.848514 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.848497    5012 scope.go:122] "RemoveContainer" containerID="0d8baa32d49b8ef323c3466c7f4543717c027a351c17d9d0eff2d592ab607fd3"
Nov 03 01:06:58.848812 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:06:58.848788    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8baa32d49b8ef323c3466c7f4543717c027a351c17d9d0eff2d592ab607fd3\": container with ID starting with 0d8baa32d49b8ef323c3466c7f4543717c027a351c17d9d0eff2d592ab607fd3 not found: ID does not exist" containerID="0d8baa32d49b8ef323c3466c7f4543717c027a351c17d9d0eff2d592ab607fd3"
Nov 03 01:06:58.848867 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.848823    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8baa32d49b8ef323c3466c7f4543717c027a351c17d9d0eff2d592ab607fd3"} err="failed to get container status \"0d8baa32d49b8ef323c3466c7f4543717c027a351c17d9d0eff2d592ab607fd3\": rpc error: code = NotFound desc = could not find container \"0d8baa32d49b8ef323c3466c7f4543717c027a351c17d9d0eff2d592ab607fd3\": container with ID starting with 0d8baa32d49b8ef323c3466c7f4543717c027a351c17d9d0eff2d592ab607fd3 not found: ID does not exist"
Nov 03 01:06:58.883385 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.883362    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"]
Nov 03 01:06:58.893457 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:58.893435    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-29d09b/test-model-29d09b-kserve-76cd55f567-qkh57"]
Nov 03 01:06:59.245586 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:06:59.245554    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8cf1798-5790-4367-aa58-d8bc7ecc0932" path="/var/lib/kubelet/pods/b8cf1798-5790-4367-aa58-d8bc7ecc0932/volumes"
Nov 03 01:07:03.967167 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:03.966885    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"]
Nov 03 01:07:07.366460 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:07:07.366405    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:07:14.296135 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:14.296101    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs"]
Nov 03 01:07:14.296673 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:14.296320    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs" podUID="b72694be-b9a8-45df-8085-c3e958c3bf31" containerName="maas-api" containerID="cri-o://5f333bede3c36f306268e5de9493b6df8cad10e5da70fdae7ff10f1105633a95" gracePeriod=30
Nov 03 01:07:14.945807 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:14.945772    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs"
Nov 03 01:07:15.006731 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:15.002416    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b72694be-b9a8-45df-8085-c3e958c3bf31-kube-api-access-j5dsd\" (UniqueName: \"kubernetes.io/projected/b72694be-b9a8-45df-8085-c3e958c3bf31-kube-api-access-j5dsd\") pod \"b72694be-b9a8-45df-8085-c3e958c3bf31\" (UID: \"b72694be-b9a8-45df-8085-c3e958c3bf31\") "
Nov 03 01:07:15.006731 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:15.002541    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b72694be-b9a8-45df-8085-c3e958c3bf31-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b72694be-b9a8-45df-8085-c3e958c3bf31-maas-api-tls\") pod \"b72694be-b9a8-45df-8085-c3e958c3bf31\" (UID: \"b72694be-b9a8-45df-8085-c3e958c3bf31\") "
Nov 03 01:07:15.006731 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:15.002692    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b72694be-b9a8-45df-8085-c3e958c3bf31-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b72694be-b9a8-45df-8085-c3e958c3bf31-openshift-service-ca\") pod \"b72694be-b9a8-45df-8085-c3e958c3bf31\" (UID: \"b72694be-b9a8-45df-8085-c3e958c3bf31\") "
Nov 03 01:07:15.006731 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:15.002795    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b72694be-b9a8-45df-8085-c3e958c3bf31-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b72694be-b9a8-45df-8085-c3e958c3bf31-metrics-certs\") pod \"b72694be-b9a8-45df-8085-c3e958c3bf31\" (UID: \"b72694be-b9a8-45df-8085-c3e958c3bf31\") "
Nov 03 01:07:15.017478 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:15.015457    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72694be-b9a8-45df-8085-c3e958c3bf31-kube-api-access-j5dsd" pod "b72694be-b9a8-45df-8085-c3e958c3bf31" (UID: "b72694be-b9a8-45df-8085-c3e958c3bf31"). InnerVolumeSpecName "kube-api-access-j5dsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:07:15.019144 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:15.019106    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b72694be-b9a8-45df-8085-c3e958c3bf31-openshift-service-ca" pod "b72694be-b9a8-45df-8085-c3e958c3bf31" (UID: "b72694be-b9a8-45df-8085-c3e958c3bf31"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:07:15.060239 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:15.051374    5012 generic.go:356] "Generic (PLEG): container finished" podID="b72694be-b9a8-45df-8085-c3e958c3bf31" containerID="5f333bede3c36f306268e5de9493b6df8cad10e5da70fdae7ff10f1105633a95" exitCode=0
Nov 03 01:07:15.060239 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:15.051691    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs" event={"ID":"b72694be-b9a8-45df-8085-c3e958c3bf31","Type":"ContainerDied","Data":"5f333bede3c36f306268e5de9493b6df8cad10e5da70fdae7ff10f1105633a95"}
Nov 03 01:07:15.060239 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:15.051823    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs" event={"ID":"b72694be-b9a8-45df-8085-c3e958c3bf31","Type":"ContainerDied","Data":"b0506e34fe58e65fb996b0d2515d0149ed3940fc19bb79f4b393eed04031b711"}
Nov 03 01:07:15.060239 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:15.051860    5012 scope.go:122] "RemoveContainer" containerID="5f333bede3c36f306268e5de9493b6df8cad10e5da70fdae7ff10f1105633a95"
Nov 03 01:07:15.060239 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:15.052071    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b72694be-b9a8-45df-8085-c3e958c3bf31-metrics-certs" pod "b72694be-b9a8-45df-8085-c3e958c3bf31" (UID: "b72694be-b9a8-45df-8085-c3e958c3bf31"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:07:15.060239 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:15.052099    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs"
Nov 03 01:07:15.060239 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:15.057609    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b72694be-b9a8-45df-8085-c3e958c3bf31-maas-api-tls" pod "b72694be-b9a8-45df-8085-c3e958c3bf31" (UID: "b72694be-b9a8-45df-8085-c3e958c3bf31"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:07:15.081843 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:15.081800    5012 scope.go:122] "RemoveContainer" containerID="5f333bede3c36f306268e5de9493b6df8cad10e5da70fdae7ff10f1105633a95"
Nov 03 01:07:15.082399 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:07:15.082317    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f333bede3c36f306268e5de9493b6df8cad10e5da70fdae7ff10f1105633a95\": container with ID starting with 5f333bede3c36f306268e5de9493b6df8cad10e5da70fdae7ff10f1105633a95 not found: ID does not exist" containerID="5f333bede3c36f306268e5de9493b6df8cad10e5da70fdae7ff10f1105633a95"
Nov 03 01:07:15.082399 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:15.082358    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f333bede3c36f306268e5de9493b6df8cad10e5da70fdae7ff10f1105633a95"} err="failed to get container status \"5f333bede3c36f306268e5de9493b6df8cad10e5da70fdae7ff10f1105633a95\": rpc error: code = NotFound desc = could not find container \"5f333bede3c36f306268e5de9493b6df8cad10e5da70fdae7ff10f1105633a95\": container with ID starting with 5f333bede3c36f306268e5de9493b6df8cad10e5da70fdae7ff10f1105633a95 not found: ID does not exist"
Nov 03 01:07:15.104361 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:15.104334    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5dsd\" (UniqueName: \"kubernetes.io/projected/b72694be-b9a8-45df-8085-c3e958c3bf31-kube-api-access-j5dsd\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:07:15.104361 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:15.104360    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b72694be-b9a8-45df-8085-c3e958c3bf31-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:07:15.104516 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:15.104371    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b72694be-b9a8-45df-8085-c3e958c3bf31-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:07:15.104516 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:15.104386    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b72694be-b9a8-45df-8085-c3e958c3bf31-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:07:15.390483 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:15.390432    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs"]
Nov 03 01:07:15.394717 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:15.394689    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-29d09b-59f4dd4688-pdvcs"]
Nov 03 01:07:17.247634 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:17.247596    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b72694be-b9a8-45df-8085-c3e958c3bf31" path="/var/lib/kubelet/pods/b72694be-b9a8-45df-8085-c3e958c3bf31/volumes"
Nov 03 01:07:18.997455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:18.997400    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc" podUID="5d60e1f1-f8dc-4863-a9db-49face095aa9" containerName="main" containerID="cri-o://be63964787c57d83f7fe641734981ada866a20cc917518c4bf4e21c57d509315" gracePeriod=45
Nov 03 01:07:19.315116 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:19.315091    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:07:19.492456 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:19.492417    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d60e1f1-f8dc-4863-a9db-49face095aa9-kube-api-access-hhlrb\" (UniqueName: \"kubernetes.io/projected/5d60e1f1-f8dc-4863-a9db-49face095aa9-kube-api-access-hhlrb\") pod \"5d60e1f1-f8dc-4863-a9db-49face095aa9\" (UID: \"5d60e1f1-f8dc-4863-a9db-49face095aa9\") "
Nov 03 01:07:19.492822 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:19.492793    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-tmp-dir\") pod \"5d60e1f1-f8dc-4863-a9db-49face095aa9\" (UID: \"5d60e1f1-f8dc-4863-a9db-49face095aa9\") "
Nov 03 01:07:19.492822 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:19.492843    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-dshm\") pod \"5d60e1f1-f8dc-4863-a9db-49face095aa9\" (UID: \"5d60e1f1-f8dc-4863-a9db-49face095aa9\") "
Nov 03 01:07:19.493126 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:19.492876    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d60e1f1-f8dc-4863-a9db-49face095aa9-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d60e1f1-f8dc-4863-a9db-49face095aa9-tls-certs\") pod \"5d60e1f1-f8dc-4863-a9db-49face095aa9\" (UID: \"5d60e1f1-f8dc-4863-a9db-49face095aa9\") "
Nov 03 01:07:19.493126 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:19.492963    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-home\" (UniqueName: \"kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-home\") pod \"5d60e1f1-f8dc-4863-a9db-49face095aa9\" (UID: \"5d60e1f1-f8dc-4863-a9db-49face095aa9\") "
Nov 03 01:07:19.493126 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:19.492999    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-model-cache\") pod \"5d60e1f1-f8dc-4863-a9db-49face095aa9\" (UID: \"5d60e1f1-f8dc-4863-a9db-49face095aa9\") "
Nov 03 01:07:19.493126 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:19.493065    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-tmp-dir" pod "5d60e1f1-f8dc-4863-a9db-49face095aa9" (UID: "5d60e1f1-f8dc-4863-a9db-49face095aa9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:07:19.493363 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:19.493240    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-home" pod "5d60e1f1-f8dc-4863-a9db-49face095aa9" (UID: "5d60e1f1-f8dc-4863-a9db-49face095aa9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:07:19.493534 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:19.493511    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-model-cache" pod "5d60e1f1-f8dc-4863-a9db-49face095aa9" (UID: "5d60e1f1-f8dc-4863-a9db-49face095aa9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:07:19.494289 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:19.494266    5012 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-tmp-dir\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:07:19.494385 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:19.494292    5012 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-home\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:07:19.494385 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:19.494307    5012 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-model-cache\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:07:19.495636 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:19.495613    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-dshm" pod "5d60e1f1-f8dc-4863-a9db-49face095aa9" (UID: "5d60e1f1-f8dc-4863-a9db-49face095aa9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:07:19.495822 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:19.495801    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d60e1f1-f8dc-4863-a9db-49face095aa9-kube-api-access-hhlrb" pod "5d60e1f1-f8dc-4863-a9db-49face095aa9" (UID: "5d60e1f1-f8dc-4863-a9db-49face095aa9"). InnerVolumeSpecName "kube-api-access-hhlrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:07:19.495903 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:19.495808    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d60e1f1-f8dc-4863-a9db-49face095aa9-tls-certs" pod "5d60e1f1-f8dc-4863-a9db-49face095aa9" (UID: "5d60e1f1-f8dc-4863-a9db-49face095aa9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:07:19.596609 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:19.596574    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhlrb\" (UniqueName: \"kubernetes.io/projected/5d60e1f1-f8dc-4863-a9db-49face095aa9-kube-api-access-hhlrb\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:07:19.596792 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:19.596616    5012 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d60e1f1-f8dc-4863-a9db-49face095aa9-dshm\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:07:19.596792 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:19.596631    5012 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d60e1f1-f8dc-4863-a9db-49face095aa9-tls-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:07:20.094720 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:20.094628    5012 generic.go:356] "Generic (PLEG): container finished" podID="5d60e1f1-f8dc-4863-a9db-49face095aa9" containerID="be63964787c57d83f7fe641734981ada866a20cc917518c4bf4e21c57d509315" exitCode=0
Nov 03 01:07:20.094720 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:20.094671    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"
Nov 03 01:07:20.094720 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:20.094703    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc" event={"ID":"5d60e1f1-f8dc-4863-a9db-49face095aa9","Type":"ContainerDied","Data":"be63964787c57d83f7fe641734981ada866a20cc917518c4bf4e21c57d509315"}
Nov 03 01:07:20.095296 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:20.094745    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc" event={"ID":"5d60e1f1-f8dc-4863-a9db-49face095aa9","Type":"ContainerDied","Data":"bf9cd8b08b444a124875423d61a81460d884db5bd877aaf635fa7740516d738b"}
Nov 03 01:07:20.095296 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:20.094762    5012 scope.go:122] "RemoveContainer" containerID="be63964787c57d83f7fe641734981ada866a20cc917518c4bf4e21c57d509315"
Nov 03 01:07:20.111456 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:20.111433    5012 scope.go:122] "RemoveContainer" containerID="be63964787c57d83f7fe641734981ada866a20cc917518c4bf4e21c57d509315"
Nov 03 01:07:20.111752 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:07:20.111728    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be63964787c57d83f7fe641734981ada866a20cc917518c4bf4e21c57d509315\": container with ID starting with be63964787c57d83f7fe641734981ada866a20cc917518c4bf4e21c57d509315 not found: ID does not exist" containerID="be63964787c57d83f7fe641734981ada866a20cc917518c4bf4e21c57d509315"
Nov 03 01:07:20.111804 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:20.111760    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be63964787c57d83f7fe641734981ada866a20cc917518c4bf4e21c57d509315"} err="failed to get container status \"be63964787c57d83f7fe641734981ada866a20cc917518c4bf4e21c57d509315\": rpc error: code = NotFound desc = could not find container \"be63964787c57d83f7fe641734981ada866a20cc917518c4bf4e21c57d509315\": container with ID starting with be63964787c57d83f7fe641734981ada866a20cc917518c4bf4e21c57d509315 not found: ID does not exist"
Nov 03 01:07:20.142176 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:20.142150    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"]
Nov 03 01:07:20.152268 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:20.152245    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc"]
Nov 03 01:07:20.269244 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:20.269204    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-537652/test-model-537652-kserve-78755f497f-p65pc" podUID="5d60e1f1-f8dc-4863-a9db-49face095aa9" containerName="main" probeResult="failure" output="Get \"https://10.132.1.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:07:21.246100 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:21.246065    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d60e1f1-f8dc-4863-a9db-49face095aa9" path="/var/lib/kubelet/pods/5d60e1f1-f8dc-4863-a9db-49face095aa9/volumes"
Nov 03 01:07:26.300874 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:26.300766    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr"]
Nov 03 01:07:26.302786 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:26.302714    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr" podUID="8ba67790-ca73-4724-b68f-21b645e1d4b3" containerName="maas-api" containerID="cri-o://d3ebc5fa4a0afc69fee7f0271cfba9e0fb1c1bef316130b1a4dee84b0c89da45" gracePeriod=30
Nov 03 01:07:27.009092 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:27.009043    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr" podUID="8ba67790-ca73-4724-b68f-21b645e1d4b3" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.1.26:8443/health\": dial tcp 10.132.1.26:8443: connect: connection refused"
Nov 03 01:07:28.074969 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:28.073922    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr"
Nov 03 01:07:28.082324 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:28.082293    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ba67790-ca73-4724-b68f-21b645e1d4b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba67790-ca73-4724-b68f-21b645e1d4b3-openshift-service-ca\") pod \"8ba67790-ca73-4724-b68f-21b645e1d4b3\" (UID: \"8ba67790-ca73-4724-b68f-21b645e1d4b3\") "
Nov 03 01:07:28.082484 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:28.082374    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ba67790-ca73-4724-b68f-21b645e1d4b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ba67790-ca73-4724-b68f-21b645e1d4b3-maas-api-tls\") pod \"8ba67790-ca73-4724-b68f-21b645e1d4b3\" (UID: \"8ba67790-ca73-4724-b68f-21b645e1d4b3\") "
Nov 03 01:07:28.082484 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:28.082412    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba67790-ca73-4724-b68f-21b645e1d4b3-kube-api-access-z5dnb\" (UniqueName: \"kubernetes.io/projected/8ba67790-ca73-4724-b68f-21b645e1d4b3-kube-api-access-z5dnb\") pod \"8ba67790-ca73-4724-b68f-21b645e1d4b3\" (UID: \"8ba67790-ca73-4724-b68f-21b645e1d4b3\") "
Nov 03 01:07:28.082614 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:28.082494    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ba67790-ca73-4724-b68f-21b645e1d4b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ba67790-ca73-4724-b68f-21b645e1d4b3-metrics-certs\") pod \"8ba67790-ca73-4724-b68f-21b645e1d4b3\" (UID: \"8ba67790-ca73-4724-b68f-21b645e1d4b3\") "
Nov 03 01:07:28.082723 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:28.082697    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba67790-ca73-4724-b68f-21b645e1d4b3-openshift-service-ca" pod "8ba67790-ca73-4724-b68f-21b645e1d4b3" (UID: "8ba67790-ca73-4724-b68f-21b645e1d4b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:07:28.083071 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:28.083050    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba67790-ca73-4724-b68f-21b645e1d4b3-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:07:28.084445 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:28.084418    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ba67790-ca73-4724-b68f-21b645e1d4b3-maas-api-tls" pod "8ba67790-ca73-4724-b68f-21b645e1d4b3" (UID: "8ba67790-ca73-4724-b68f-21b645e1d4b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:07:28.084520 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:28.084441    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ba67790-ca73-4724-b68f-21b645e1d4b3-metrics-certs" pod "8ba67790-ca73-4724-b68f-21b645e1d4b3" (UID: "8ba67790-ca73-4724-b68f-21b645e1d4b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:07:28.084520 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:28.084506    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba67790-ca73-4724-b68f-21b645e1d4b3-kube-api-access-z5dnb" pod "8ba67790-ca73-4724-b68f-21b645e1d4b3" (UID: "8ba67790-ca73-4724-b68f-21b645e1d4b3"). InnerVolumeSpecName "kube-api-access-z5dnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:07:28.184344 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:28.184301    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ba67790-ca73-4724-b68f-21b645e1d4b3-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:07:28.184344 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:28.184336    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5dnb\" (UniqueName: \"kubernetes.io/projected/8ba67790-ca73-4724-b68f-21b645e1d4b3-kube-api-access-z5dnb\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:07:28.184344 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:28.184349    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ba67790-ca73-4724-b68f-21b645e1d4b3-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:07:28.193062 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:28.193026    5012 generic.go:356] "Generic (PLEG): container finished" podID="8ba67790-ca73-4724-b68f-21b645e1d4b3" containerID="d3ebc5fa4a0afc69fee7f0271cfba9e0fb1c1bef316130b1a4dee84b0c89da45" exitCode=0
Nov 03 01:07:28.193554 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:28.193344    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr"
Nov 03 01:07:28.196450 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:28.193365    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr" event={"ID":"8ba67790-ca73-4724-b68f-21b645e1d4b3","Type":"ContainerDied","Data":"d3ebc5fa4a0afc69fee7f0271cfba9e0fb1c1bef316130b1a4dee84b0c89da45"}
Nov 03 01:07:28.196450 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:28.196204    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr" event={"ID":"8ba67790-ca73-4724-b68f-21b645e1d4b3","Type":"ContainerDied","Data":"a71280b575fee0110ae970e398dd00b590521a813e7c45fbceeb27a78fa61631"}
Nov 03 01:07:28.196450 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:28.196240    5012 scope.go:122] "RemoveContainer" containerID="d3ebc5fa4a0afc69fee7f0271cfba9e0fb1c1bef316130b1a4dee84b0c89da45"
Nov 03 01:07:28.362001 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:28.346976    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr"]
Nov 03 01:07:28.381539 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:28.380902    5012 scope.go:122] "RemoveContainer" containerID="d3ebc5fa4a0afc69fee7f0271cfba9e0fb1c1bef316130b1a4dee84b0c89da45"
Nov 03 01:07:28.381539 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:07:28.381494    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3ebc5fa4a0afc69fee7f0271cfba9e0fb1c1bef316130b1a4dee84b0c89da45\": container with ID starting with d3ebc5fa4a0afc69fee7f0271cfba9e0fb1c1bef316130b1a4dee84b0c89da45 not found: ID does not exist" containerID="d3ebc5fa4a0afc69fee7f0271cfba9e0fb1c1bef316130b1a4dee84b0c89da45"
Nov 03 01:07:28.381683 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:28.381541    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3ebc5fa4a0afc69fee7f0271cfba9e0fb1c1bef316130b1a4dee84b0c89da45"} err="failed to get container status \"d3ebc5fa4a0afc69fee7f0271cfba9e0fb1c1bef316130b1a4dee84b0c89da45\": rpc error: code = NotFound desc = could not find container \"d3ebc5fa4a0afc69fee7f0271cfba9e0fb1c1bef316130b1a4dee84b0c89da45\": container with ID starting with d3ebc5fa4a0afc69fee7f0271cfba9e0fb1c1bef316130b1a4dee84b0c89da45 not found: ID does not exist"
Nov 03 01:07:28.424943 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:28.413687    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-70c7d6-597f98c89d-xjzkr"]
Nov 03 01:07:29.257431 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:07:29.257397    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ba67790-ca73-4724-b68f-21b645e1d4b3" path="/var/lib/kubelet/pods/8ba67790-ca73-4724-b68f-21b645e1d4b3/volumes"
Nov 03 01:07:37.364474 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:07:37.364435    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:08:07.364698 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:08:07.364658    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:08:37.363365 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:08:37.363325    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:08:49.765063 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.765025    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j"]
Nov 03 01:08:49.765902 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.765878    5012 cpu_manager.go:414] "containerMap: removing container" podUID="72d32ad0-f7d6-4c17-8781-83d65335f0e6" containerName="main"
Nov 03 01:08:49.765902 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.765904    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72d32ad0-f7d6-4c17-8781-83d65335f0e6" containerName="main"
Nov 03 01:08:49.765902 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.765933    5012 cpu_manager.go:414] "containerMap: removing container" podUID="b8cf1798-5790-4367-aa58-d8bc7ecc0932" containerName="main"
Nov 03 01:08:49.766147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.765942    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8cf1798-5790-4367-aa58-d8bc7ecc0932" containerName="main"
Nov 03 01:08:49.766147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.765962    5012 cpu_manager.go:414] "containerMap: removing container" podUID="8ba67790-ca73-4724-b68f-21b645e1d4b3" containerName="maas-api"
Nov 03 01:08:49.766147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.765971    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba67790-ca73-4724-b68f-21b645e1d4b3" containerName="maas-api"
Nov 03 01:08:49.766147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.765991    5012 cpu_manager.go:414] "containerMap: removing container" podUID="209af59a-bfd0-4aab-b2bf-7c789190f57e" containerName="istio-proxy"
Nov 03 01:08:49.766147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.765996    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="209af59a-bfd0-4aab-b2bf-7c789190f57e" containerName="istio-proxy"
Nov 03 01:08:49.766147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766002    5012 cpu_manager.go:414] "containerMap: removing container" podUID="4233c829-fa1c-443e-b7b6-9522c977fa98" containerName="istio-proxy"
Nov 03 01:08:49.766147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766007    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4233c829-fa1c-443e-b7b6-9522c977fa98" containerName="istio-proxy"
Nov 03 01:08:49.766147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766015    5012 cpu_manager.go:414] "containerMap: removing container" podUID="209e0827-a624-4090-bcc4-4ac72c7eb1e6" containerName="main"
Nov 03 01:08:49.766147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766019    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="209e0827-a624-4090-bcc4-4ac72c7eb1e6" containerName="main"
Nov 03 01:08:49.766147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766030    5012 cpu_manager.go:414] "containerMap: removing container" podUID="8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed" containerName="maas-api"
Nov 03 01:08:49.766147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766035    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed" containerName="maas-api"
Nov 03 01:08:49.766147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766046    5012 cpu_manager.go:414] "containerMap: removing container" podUID="4d5d2f80-69a4-45d5-90e1-2a8335e38cda" containerName="maas-api"
Nov 03 01:08:49.766147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766053    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d5d2f80-69a4-45d5-90e1-2a8335e38cda" containerName="maas-api"
Nov 03 01:08:49.766147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766067    5012 cpu_manager.go:414] "containerMap: removing container" podUID="56c4471b-fbec-4f61-8db4-3c7f2c7298c6" containerName="istio-proxy"
Nov 03 01:08:49.766147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766073    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56c4471b-fbec-4f61-8db4-3c7f2c7298c6" containerName="istio-proxy"
Nov 03 01:08:49.766147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766085    5012 cpu_manager.go:414] "containerMap: removing container" podUID="b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3" containerName="payload-pre-processing"
Nov 03 01:08:49.766147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766092    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3" containerName="payload-pre-processing"
Nov 03 01:08:49.766147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766105    5012 cpu_manager.go:414] "containerMap: removing container" podUID="5d60e1f1-f8dc-4863-a9db-49face095aa9" containerName="main"
Nov 03 01:08:49.766147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766113    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d60e1f1-f8dc-4863-a9db-49face095aa9" containerName="main"
Nov 03 01:08:49.766147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766127    5012 cpu_manager.go:414] "containerMap: removing container" podUID="b72694be-b9a8-45df-8085-c3e958c3bf31" containerName="maas-api"
Nov 03 01:08:49.766147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766134    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b72694be-b9a8-45df-8085-c3e958c3bf31" containerName="maas-api"
Nov 03 01:08:49.766147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766151    5012 cpu_manager.go:414] "containerMap: removing container" podUID="876fffce-7b9b-4aeb-8437-4c60987a5056" containerName="istio-proxy"
Nov 03 01:08:49.766147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766156    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="876fffce-7b9b-4aeb-8437-4c60987a5056" containerName="istio-proxy"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766167    5012 cpu_manager.go:414] "containerMap: removing container" podUID="0b949944-2ed2-4a75-82d1-a2e2854e3355" containerName="istio-proxy"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766172    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b949944-2ed2-4a75-82d1-a2e2854e3355" containerName="istio-proxy"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766181    5012 cpu_manager.go:414] "containerMap: removing container" podUID="d79c28c1-9ce7-4040-90a7-6a423b1e349e" containerName="istio-proxy"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766185    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d79c28c1-9ce7-4040-90a7-6a423b1e349e" containerName="istio-proxy"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766192    5012 cpu_manager.go:414] "containerMap: removing container" podUID="77f5e43c-70c1-4f1b-ac62-412c0939ecbc" containerName="maas-api"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766197    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f5e43c-70c1-4f1b-ac62-412c0939ecbc" containerName="maas-api"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766212    5012 cpu_manager.go:414] "containerMap: removing container" podUID="209af59a-bfd0-4aab-b2bf-7c789190f57e" containerName="istio-proxy"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766216    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="209af59a-bfd0-4aab-b2bf-7c789190f57e" containerName="istio-proxy"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766231    5012 cpu_manager.go:414] "containerMap: removing container" podUID="a5e7d5fd-29e5-4719-8239-922e7a3aeeed" containerName="istio-proxy"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766239    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e7d5fd-29e5-4719-8239-922e7a3aeeed" containerName="istio-proxy"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766253    5012 cpu_manager.go:414] "containerMap: removing container" podUID="8216aa2d-f296-4d63-ad6b-62e2aa583c5c" containerName="payload-processing"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766258    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8216aa2d-f296-4d63-ad6b-62e2aa583c5c" containerName="payload-processing"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766416    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="8216aa2d-f296-4d63-ad6b-62e2aa583c5c" containerName="payload-processing"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766430    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="72d32ad0-f7d6-4c17-8781-83d65335f0e6" containerName="main"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766438    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8cf1798-5790-4367-aa58-d8bc7ecc0932" containerName="main"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766446    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba67790-ca73-4724-b68f-21b645e1d4b3" containerName="maas-api"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766455    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="209af59a-bfd0-4aab-b2bf-7c789190f57e" containerName="istio-proxy"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766463    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="209e0827-a624-4090-bcc4-4ac72c7eb1e6" containerName="main"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766479    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="4233c829-fa1c-443e-b7b6-9522c977fa98" containerName="istio-proxy"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766489    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f69a68b-77af-47ad-b4d5-bcd2cc02c1ed" containerName="maas-api"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766497    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d5d2f80-69a4-45d5-90e1-2a8335e38cda" containerName="maas-api"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766503    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="56c4471b-fbec-4f61-8db4-3c7f2c7298c6" containerName="istio-proxy"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766507    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8cf0370-bd15-4ea1-9aa2-e83ce4938ea3" containerName="payload-pre-processing"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766512    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d60e1f1-f8dc-4863-a9db-49face095aa9" containerName="main"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766519    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="b72694be-b9a8-45df-8085-c3e958c3bf31" containerName="maas-api"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766528    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="876fffce-7b9b-4aeb-8437-4c60987a5056" containerName="istio-proxy"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766538    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b949944-2ed2-4a75-82d1-a2e2854e3355" containerName="istio-proxy"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766549    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="d79c28c1-9ce7-4040-90a7-6a423b1e349e" containerName="istio-proxy"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766555    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="77f5e43c-70c1-4f1b-ac62-412c0939ecbc" containerName="maas-api"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766565    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="4233c829-fa1c-443e-b7b6-9522c977fa98" containerName="istio-proxy"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766571    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b949944-2ed2-4a75-82d1-a2e2854e3355" containerName="istio-proxy"
Nov 03 01:08:49.767218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.766578    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e7d5fd-29e5-4719-8239-922e7a3aeeed" containerName="istio-proxy"
Nov 03 01:08:49.770606 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.770577    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j"
Nov 03 01:08:49.787543 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.787522    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c6e808d7-serving-cert\""
Nov 03 01:08:49.815433 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.815403    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j"]
Nov 03 01:08:49.873885 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.873849    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-openshift-service-ca\") pod \"maas-api-e2e-mt-c6e808d7-59868d594f-th79j\" (UID: \"16461b4f-b9e0-4313-afbd-15c34c4bd9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j"
Nov 03 01:08:49.873885 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.873890    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-maas-api-tls\") pod \"maas-api-e2e-mt-c6e808d7-59868d594f-th79j\" (UID: \"16461b4f-b9e0-4313-afbd-15c34c4bd9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j"
Nov 03 01:08:49.874162 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.874080    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5wmq\" (UniqueName: \"kubernetes.io/projected/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-kube-api-access-d5wmq\") pod \"maas-api-e2e-mt-c6e808d7-59868d594f-th79j\" (UID: \"16461b4f-b9e0-4313-afbd-15c34c4bd9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j"
Nov 03 01:08:49.874317 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.874294    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-metrics-certs\") pod \"maas-api-e2e-mt-c6e808d7-59868d594f-th79j\" (UID: \"16461b4f-b9e0-4313-afbd-15c34c4bd9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j"
Nov 03 01:08:49.983643 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.983096    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-metrics-certs\") pod \"maas-api-e2e-mt-c6e808d7-59868d594f-th79j\" (UID: \"16461b4f-b9e0-4313-afbd-15c34c4bd9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j"
Nov 03 01:08:49.983643 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.983195    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-openshift-service-ca\") pod \"maas-api-e2e-mt-c6e808d7-59868d594f-th79j\" (UID: \"16461b4f-b9e0-4313-afbd-15c34c4bd9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j"
Nov 03 01:08:49.983643 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.983226    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-maas-api-tls\") pod \"maas-api-e2e-mt-c6e808d7-59868d594f-th79j\" (UID: \"16461b4f-b9e0-4313-afbd-15c34c4bd9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j"
Nov 03 01:08:49.983643 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.983310    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5wmq\" (UniqueName: \"kubernetes.io/projected/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-kube-api-access-d5wmq\") pod \"maas-api-e2e-mt-c6e808d7-59868d594f-th79j\" (UID: \"16461b4f-b9e0-4313-afbd-15c34c4bd9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j"
Nov 03 01:08:49.991840 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.991806    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-openshift-service-ca\") pod \"maas-api-e2e-mt-c6e808d7-59868d594f-th79j\" (UID: \"16461b4f-b9e0-4313-afbd-15c34c4bd9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j"
Nov 03 01:08:49.999533 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:49.999492    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-metrics-certs\") pod \"maas-api-e2e-mt-c6e808d7-59868d594f-th79j\" (UID: \"16461b4f-b9e0-4313-afbd-15c34c4bd9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j"
Nov 03 01:08:50.006576 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:50.006544    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5wmq\" (UniqueName: \"kubernetes.io/projected/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-kube-api-access-d5wmq\") pod \"maas-api-e2e-mt-c6e808d7-59868d594f-th79j\" (UID: \"16461b4f-b9e0-4313-afbd-15c34c4bd9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j"
Nov 03 01:08:50.008439 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:50.008404    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-maas-api-tls\") pod \"maas-api-e2e-mt-c6e808d7-59868d594f-th79j\" (UID: \"16461b4f-b9e0-4313-afbd-15c34c4bd9dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j"
Nov 03 01:08:50.089606 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:50.089529    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j"
Nov 03 01:08:50.397828 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:50.397796    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j"]
Nov 03 01:08:50.398312 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:08:50.398291    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16461b4f_b9e0_4313_afbd_15c34c4bd9dd.slice/crio-9777c50e67e58fbeb1182dcd1a8530514e599c269cb03f2e0e762a726b04b364 WatchSource:0}: Error finding container 9777c50e67e58fbeb1182dcd1a8530514e599c269cb03f2e0e762a726b04b364: Status 404 returned error can't find the container with id 9777c50e67e58fbeb1182dcd1a8530514e599c269cb03f2e0e762a726b04b364
Nov 03 01:08:50.399986 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:50.399965    5012 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 01:08:51.016971 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:51.016882    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j" event={"ID":"16461b4f-b9e0-4313-afbd-15c34c4bd9dd","Type":"ContainerStarted","Data":"6f02f61a6066cfd9e4aafeb131391680ff6e66db2debdcac000ee3baf8fc4e70"}
Nov 03 01:08:51.017363 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:51.016990    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j"
Nov 03 01:08:51.017363 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:51.017018    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j" event={"ID":"16461b4f-b9e0-4313-afbd-15c34c4bd9dd","Type":"ContainerStarted","Data":"9777c50e67e58fbeb1182dcd1a8530514e599c269cb03f2e0e762a726b04b364"}
Nov 03 01:08:51.060228 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:51.060170    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j" podStartSLOduration=1.7849448209999998 podStartE2EDuration="2.060156161s" totalImagesPullingTime="275.21134ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:08:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:08:51.055567414 +0000 UTC m=+3813.935151458" watchObservedRunningTime="2026-11-03 01:08:51.060156161 +0000 UTC m=+3813.939740204"
Nov 03 01:08:57.030499 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:08:57.030458    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j"
Nov 03 01:09:07.363884 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:09:07.363795    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:09:37.363318 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:09:37.363276    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:09:58.883480 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:09:58.883442    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f"]
Nov 03 01:09:58.884110 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:09:58.883702    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f" podUID="ce10e6fa-6b29-4edc-ac96-e81e7c7eebab" containerName="maas-api" containerID="cri-o://c8273d6e9ba441aeb7b8203f44632a940a1b17b8b0844d66c80fd868bd269bd2" gracePeriod=30
Nov 03 01:09:59.219313 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:09:59.219280    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f"
Nov 03 01:09:59.232278 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:09:59.232250    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-openshift-service-ca\") pod \"ce10e6fa-6b29-4edc-ac96-e81e7c7eebab\" (UID: \"ce10e6fa-6b29-4edc-ac96-e81e7c7eebab\") "
Nov 03 01:09:59.232448 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:09:59.232339    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-maas-api-tls\") pod \"ce10e6fa-6b29-4edc-ac96-e81e7c7eebab\" (UID: \"ce10e6fa-6b29-4edc-ac96-e81e7c7eebab\") "
Nov 03 01:09:59.232448 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:09:59.232396    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-metrics-certs\") pod \"ce10e6fa-6b29-4edc-ac96-e81e7c7eebab\" (UID: \"ce10e6fa-6b29-4edc-ac96-e81e7c7eebab\") "
Nov 03 01:09:59.232627 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:09:59.232610    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-kube-api-access-d6h98\" (UniqueName: \"kubernetes.io/projected/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-kube-api-access-d6h98\") pod \"ce10e6fa-6b29-4edc-ac96-e81e7c7eebab\" (UID: \"ce10e6fa-6b29-4edc-ac96-e81e7c7eebab\") "
Nov 03 01:09:59.232699 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:09:59.232625    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-openshift-service-ca" pod "ce10e6fa-6b29-4edc-ac96-e81e7c7eebab" (UID: "ce10e6fa-6b29-4edc-ac96-e81e7c7eebab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:09:59.233238 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:09:59.233213    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:09:59.234695 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:09:59.234669    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-kube-api-access-d6h98" pod "ce10e6fa-6b29-4edc-ac96-e81e7c7eebab" (UID: "ce10e6fa-6b29-4edc-ac96-e81e7c7eebab"). InnerVolumeSpecName "kube-api-access-d6h98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:09:59.234998 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:09:59.234974    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-metrics-certs" pod "ce10e6fa-6b29-4edc-ac96-e81e7c7eebab" (UID: "ce10e6fa-6b29-4edc-ac96-e81e7c7eebab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:09:59.235095 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:09:59.235009    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-maas-api-tls" pod "ce10e6fa-6b29-4edc-ac96-e81e7c7eebab" (UID: "ce10e6fa-6b29-4edc-ac96-e81e7c7eebab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:09:59.410683 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:09:59.410650    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:09:59.411070 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:09:59.411046    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:09:59.411230 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:09:59.411215    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6h98\" (UniqueName: \"kubernetes.io/projected/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab-kube-api-access-d6h98\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:09:59.567270 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:09:59.567196    5012 generic.go:356] "Generic (PLEG): container finished" podID="ce10e6fa-6b29-4edc-ac96-e81e7c7eebab" containerID="c8273d6e9ba441aeb7b8203f44632a940a1b17b8b0844d66c80fd868bd269bd2" exitCode=0
Nov 03 01:09:59.567403 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:09:59.567298    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f"
Nov 03 01:09:59.567403 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:09:59.567317    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f" event={"ID":"ce10e6fa-6b29-4edc-ac96-e81e7c7eebab","Type":"ContainerDied","Data":"c8273d6e9ba441aeb7b8203f44632a940a1b17b8b0844d66c80fd868bd269bd2"}
Nov 03 01:09:59.567403 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:09:59.567358    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f" event={"ID":"ce10e6fa-6b29-4edc-ac96-e81e7c7eebab","Type":"ContainerDied","Data":"bdd8addd757a49ea59b4e6835843fd92b1811e19b1325e700645621a3aebd478"}
Nov 03 01:09:59.567403 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:09:59.567378    5012 scope.go:122] "RemoveContainer" containerID="c8273d6e9ba441aeb7b8203f44632a940a1b17b8b0844d66c80fd868bd269bd2"
Nov 03 01:09:59.581234 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:09:59.581212    5012 scope.go:122] "RemoveContainer" containerID="c8273d6e9ba441aeb7b8203f44632a940a1b17b8b0844d66c80fd868bd269bd2"
Nov 03 01:09:59.581518 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:09:59.581498    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8273d6e9ba441aeb7b8203f44632a940a1b17b8b0844d66c80fd868bd269bd2\": container with ID starting with c8273d6e9ba441aeb7b8203f44632a940a1b17b8b0844d66c80fd868bd269bd2 not found: ID does not exist" containerID="c8273d6e9ba441aeb7b8203f44632a940a1b17b8b0844d66c80fd868bd269bd2"
Nov 03 01:09:59.581573 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:09:59.581525    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8273d6e9ba441aeb7b8203f44632a940a1b17b8b0844d66c80fd868bd269bd2"} err="failed to get container status \"c8273d6e9ba441aeb7b8203f44632a940a1b17b8b0844d66c80fd868bd269bd2\": rpc error: code = NotFound desc = could not find container \"c8273d6e9ba441aeb7b8203f44632a940a1b17b8b0844d66c80fd868bd269bd2\": container with ID starting with c8273d6e9ba441aeb7b8203f44632a940a1b17b8b0844d66c80fd868bd269bd2 not found: ID does not exist"
Nov 03 01:09:59.615105 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:09:59.615081    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f"]
Nov 03 01:09:59.625687 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:09:59.625665    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d87781-67899cf7ff-dcp5f"]
Nov 03 01:10:01.245520 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:01.245482    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce10e6fa-6b29-4edc-ac96-e81e7c7eebab" path="/var/lib/kubelet/pods/ce10e6fa-6b29-4edc-ac96-e81e7c7eebab/volumes"
Nov 03 01:10:07.362653 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:10:07.362610    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:10:10.112856 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:10.112818    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk"]
Nov 03 01:10:10.113383 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:10.113150    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk" podUID="2a72ca88-0371-4fd7-ab82-3019bd21d9f6" containerName="maas-api" containerID="cri-o://02fb35688ea869c27c7723f531fe80e0411dff0f0da115c363fb005320116933" gracePeriod=30
Nov 03 01:10:10.435437 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:10.435412    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk"
Nov 03 01:10:10.561566 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:10.561533    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-openshift-service-ca\") pod \"2a72ca88-0371-4fd7-ab82-3019bd21d9f6\" (UID: \"2a72ca88-0371-4fd7-ab82-3019bd21d9f6\") "
Nov 03 01:10:10.561725 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:10.561591    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-metrics-certs\") pod \"2a72ca88-0371-4fd7-ab82-3019bd21d9f6\" (UID: \"2a72ca88-0371-4fd7-ab82-3019bd21d9f6\") "
Nov 03 01:10:10.561809 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:10.561790    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-maas-api-tls\") pod \"2a72ca88-0371-4fd7-ab82-3019bd21d9f6\" (UID: \"2a72ca88-0371-4fd7-ab82-3019bd21d9f6\") "
Nov 03 01:10:10.561883 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:10.561867    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-kube-api-access-j4zx7\" (UniqueName: \"kubernetes.io/projected/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-kube-api-access-j4zx7\") pod \"2a72ca88-0371-4fd7-ab82-3019bd21d9f6\" (UID: \"2a72ca88-0371-4fd7-ab82-3019bd21d9f6\") "
Nov 03 01:10:10.561981 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:10.561873    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-openshift-service-ca" pod "2a72ca88-0371-4fd7-ab82-3019bd21d9f6" (UID: "2a72ca88-0371-4fd7-ab82-3019bd21d9f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:10:10.562301 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:10.562275    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:10:10.563887 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:10.563859    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-metrics-certs" pod "2a72ca88-0371-4fd7-ab82-3019bd21d9f6" (UID: "2a72ca88-0371-4fd7-ab82-3019bd21d9f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:10:10.564013 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:10.563884    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-maas-api-tls" pod "2a72ca88-0371-4fd7-ab82-3019bd21d9f6" (UID: "2a72ca88-0371-4fd7-ab82-3019bd21d9f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:10:10.564013 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:10.563937    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-kube-api-access-j4zx7" pod "2a72ca88-0371-4fd7-ab82-3019bd21d9f6" (UID: "2a72ca88-0371-4fd7-ab82-3019bd21d9f6"). InnerVolumeSpecName "kube-api-access-j4zx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:10:10.671990 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:10.666715    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:10:10.671990 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:10.666752    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:10:10.671990 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:10.666766    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4zx7\" (UniqueName: \"kubernetes.io/projected/2a72ca88-0371-4fd7-ab82-3019bd21d9f6-kube-api-access-j4zx7\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:10:10.727220 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:10.727184    5012 generic.go:356] "Generic (PLEG): container finished" podID="2a72ca88-0371-4fd7-ab82-3019bd21d9f6" containerID="02fb35688ea869c27c7723f531fe80e0411dff0f0da115c363fb005320116933" exitCode=0
Nov 03 01:10:10.727405 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:10.727235    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk" event={"ID":"2a72ca88-0371-4fd7-ab82-3019bd21d9f6","Type":"ContainerDied","Data":"02fb35688ea869c27c7723f531fe80e0411dff0f0da115c363fb005320116933"}
Nov 03 01:10:10.727405 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:10.727296    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk" event={"ID":"2a72ca88-0371-4fd7-ab82-3019bd21d9f6","Type":"ContainerDied","Data":"42eefa4c7a59962ccc5f596b64b3209c0f0803663cb40c002a87b01565dc055d"}
Nov 03 01:10:10.727405 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:10.727321    5012 scope.go:122] "RemoveContainer" containerID="02fb35688ea869c27c7723f531fe80e0411dff0f0da115c363fb005320116933"
Nov 03 01:10:10.727405 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:10.727240    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk"
Nov 03 01:10:10.740225 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:10.740193    5012 scope.go:122] "RemoveContainer" containerID="02fb35688ea869c27c7723f531fe80e0411dff0f0da115c363fb005320116933"
Nov 03 01:10:10.740525 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:10:10.740492    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02fb35688ea869c27c7723f531fe80e0411dff0f0da115c363fb005320116933\": container with ID starting with 02fb35688ea869c27c7723f531fe80e0411dff0f0da115c363fb005320116933 not found: ID does not exist" containerID="02fb35688ea869c27c7723f531fe80e0411dff0f0da115c363fb005320116933"
Nov 03 01:10:10.740652 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:10.740532    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02fb35688ea869c27c7723f531fe80e0411dff0f0da115c363fb005320116933"} err="failed to get container status \"02fb35688ea869c27c7723f531fe80e0411dff0f0da115c363fb005320116933\": rpc error: code = NotFound desc = could not find container \"02fb35688ea869c27c7723f531fe80e0411dff0f0da115c363fb005320116933\": container with ID starting with 02fb35688ea869c27c7723f531fe80e0411dff0f0da115c363fb005320116933 not found: ID does not exist"
Nov 03 01:10:10.776024 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:10.775998    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk"]
Nov 03 01:10:10.786224 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:10.786203    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d4d8c2-7fd995d856-fcctk"]
Nov 03 01:10:11.245283 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:11.245246    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a72ca88-0371-4fd7-ab82-3019bd21d9f6" path="/var/lib/kubelet/pods/2a72ca88-0371-4fd7-ab82-3019bd21d9f6/volumes"
Nov 03 01:10:18.563384 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:18.563351    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 01:10:18.568523 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:18.568497    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 01:10:18.569940 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:18.569896    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 01:10:18.574599 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:10:18.574582    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 01:10:37.363383 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:10:37.363297    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:11:07.363338 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:11:07.363299    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:11:14.062115 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:14.062080    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx"]
Nov 03 01:11:14.062652 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:14.062300    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx" podUID="2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64" containerName="maas-api" containerID="cri-o://971b09c4aa28d778da4894db6dbb2cf297f826c8efb336623357b4858990a921" gracePeriod=30
Nov 03 01:11:14.265879 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:14.265846    5012 generic.go:356] "Generic (PLEG): container finished" podID="2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64" containerID="971b09c4aa28d778da4894db6dbb2cf297f826c8efb336623357b4858990a921" exitCode=0
Nov 03 01:11:14.266074 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:14.265932    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx" event={"ID":"2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64","Type":"ContainerDied","Data":"971b09c4aa28d778da4894db6dbb2cf297f826c8efb336623357b4858990a921"}
Nov 03 01:11:14.402597 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:14.402568    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx"
Nov 03 01:11:14.461993 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:14.461370    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-openshift-service-ca\") pod \"2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64\" (UID: \"2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64\") "
Nov 03 01:11:14.461993 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:14.461443    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-metrics-certs\") pod \"2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64\" (UID: \"2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64\") "
Nov 03 01:11:14.461993 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:14.461503    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-kube-api-access-wcxft\" (UniqueName: \"kubernetes.io/projected/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-kube-api-access-wcxft\") pod \"2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64\" (UID: \"2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64\") "
Nov 03 01:11:14.461993 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:14.461603    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-maas-api-tls\") pod \"2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64\" (UID: \"2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64\") "
Nov 03 01:11:14.463968 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:14.463892    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-openshift-service-ca" pod "2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64" (UID: "2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:11:14.466869 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:14.466159    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-maas-api-tls" pod "2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64" (UID: "2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:11:14.478125 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:14.478095    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-kube-api-access-wcxft" pod "2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64" (UID: "2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64"). InnerVolumeSpecName "kube-api-access-wcxft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:14.479996 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:14.479873    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-metrics-certs" pod "2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64" (UID: "2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:11:14.583931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:14.564837    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:14.583931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:14.564878    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:14.583931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:14.564894    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcxft\" (UniqueName: \"kubernetes.io/projected/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-kube-api-access-wcxft\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:14.583931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:14.564920    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:15.287060 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:15.287025    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx"
Nov 03 01:11:15.287532 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:15.287051    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx" event={"ID":"2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64","Type":"ContainerDied","Data":"acc9bfc32704d824872fb1dc126be9482642a2b98fe26dfee0adb55bda78bcb5"}
Nov 03 01:11:15.287532 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:15.287144    5012 scope.go:122] "RemoveContainer" containerID="971b09c4aa28d778da4894db6dbb2cf297f826c8efb336623357b4858990a921"
Nov 03 01:11:15.337434 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:15.337371    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx"]
Nov 03 01:11:15.347284 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:15.347261    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-909268-6664f79f49-tv2bx"]
Nov 03 01:11:17.245866 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:17.245828    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64" path="/var/lib/kubelet/pods/2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64/volumes"
Nov 03 01:11:31.669274 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:31.669234    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd"]
Nov 03 01:11:35.383117 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:35.383082    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"]
Nov 03 01:11:35.410100 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:35.410067    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"]
Nov 03 01:11:36.670270 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:36.670210    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd" podUID="5d0c60ba-1ab0-4a70-82b9-78580583e5cf" containerName="payload-processing" containerID="cri-o://119c0b800c3fa39ed88d4b22b3a430413422c2eb25665093cd221ef745d67a37" gracePeriod=25
Nov 03 01:11:37.363324 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:11:37.363285    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:11:39.464796 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:39.464760    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j"]
Nov 03 01:11:39.465234 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:39.464962    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j" podUID="16461b4f-b9e0-4313-afbd-15c34c4bd9dd" containerName="maas-api" containerID="cri-o://6f02f61a6066cfd9e4aafeb131391680ff6e66db2debdcac000ee3baf8fc4e70" gracePeriod=30
Nov 03 01:11:39.622571 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:39.622539    5012 generic.go:356] "Generic (PLEG): container finished" podID="16461b4f-b9e0-4313-afbd-15c34c4bd9dd" containerID="6f02f61a6066cfd9e4aafeb131391680ff6e66db2debdcac000ee3baf8fc4e70" exitCode=0
Nov 03 01:11:39.622740 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:39.622633    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j" event={"ID":"16461b4f-b9e0-4313-afbd-15c34c4bd9dd","Type":"ContainerDied","Data":"6f02f61a6066cfd9e4aafeb131391680ff6e66db2debdcac000ee3baf8fc4e70"}
Nov 03 01:11:39.787005 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:39.786979    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j"
Nov 03 01:11:40.027978 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:40.027883    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-maas-api-tls\") pod \"16461b4f-b9e0-4313-afbd-15c34c4bd9dd\" (UID: \"16461b4f-b9e0-4313-afbd-15c34c4bd9dd\") "
Nov 03 01:11:40.027978 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:40.027934    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-kube-api-access-d5wmq\" (UniqueName: \"kubernetes.io/projected/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-kube-api-access-d5wmq\") pod \"16461b4f-b9e0-4313-afbd-15c34c4bd9dd\" (UID: \"16461b4f-b9e0-4313-afbd-15c34c4bd9dd\") "
Nov 03 01:11:40.027978 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:40.027976    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-metrics-certs\") pod \"16461b4f-b9e0-4313-afbd-15c34c4bd9dd\" (UID: \"16461b4f-b9e0-4313-afbd-15c34c4bd9dd\") "
Nov 03 01:11:40.028306 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:40.028038    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-openshift-service-ca\") pod \"16461b4f-b9e0-4313-afbd-15c34c4bd9dd\" (UID: \"16461b4f-b9e0-4313-afbd-15c34c4bd9dd\") "
Nov 03 01:11:40.028455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:40.028426    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-openshift-service-ca" pod "16461b4f-b9e0-4313-afbd-15c34c4bd9dd" (UID: "16461b4f-b9e0-4313-afbd-15c34c4bd9dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:11:40.028594 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:40.028566    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:40.030170 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:40.030139    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-maas-api-tls" pod "16461b4f-b9e0-4313-afbd-15c34c4bd9dd" (UID: "16461b4f-b9e0-4313-afbd-15c34c4bd9dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:11:40.030268 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:40.030167    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-metrics-certs" pod "16461b4f-b9e0-4313-afbd-15c34c4bd9dd" (UID: "16461b4f-b9e0-4313-afbd-15c34c4bd9dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:11:40.030268 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:40.030176    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-kube-api-access-d5wmq" pod "16461b4f-b9e0-4313-afbd-15c34c4bd9dd" (UID: "16461b4f-b9e0-4313-afbd-15c34c4bd9dd"). InnerVolumeSpecName "kube-api-access-d5wmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:40.129234 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:40.129207    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:40.129234 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:40.129232    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5wmq\" (UniqueName: \"kubernetes.io/projected/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-kube-api-access-d5wmq\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:40.129370 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:40.129244    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16461b4f-b9e0-4313-afbd-15c34c4bd9dd-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:40.632141 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:40.632113    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j"
Nov 03 01:11:40.632631 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:40.632116    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j" event={"ID":"16461b4f-b9e0-4313-afbd-15c34c4bd9dd","Type":"ContainerDied","Data":"9777c50e67e58fbeb1182dcd1a8530514e599c269cb03f2e0e762a726b04b364"}
Nov 03 01:11:40.632631 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:40.632244    5012 scope.go:122] "RemoveContainer" containerID="6f02f61a6066cfd9e4aafeb131391680ff6e66db2debdcac000ee3baf8fc4e70"
Nov 03 01:11:40.679253 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:40.679229    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j"]
Nov 03 01:11:40.689355 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:40.689333    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6e808d7-59868d594f-th79j"]
Nov 03 01:11:41.245113 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:41.245080    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16461b4f-b9e0-4313-afbd-15c34c4bd9dd" path="/var/lib/kubelet/pods/16461b4f-b9e0-4313-afbd-15c34c4bd9dd/volumes"
Nov 03 01:11:44.749393 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:44.749357    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd" podUID="5d0c60ba-1ab0-4a70-82b9-78580583e5cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:11:46.559691 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:46.559662    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx"]
Nov 03 01:11:46.595772 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:46.595741    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8"]
Nov 03 01:11:50.405432 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.405384    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main" containerID="cri-o://c71c160d763be7b46f3e1c8cda3c3a14c6897d9c0fd32d7f10dbbcf47cc47d20" gracePeriod=45
Nov 03 01:11:50.430617 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.430568    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main" containerID="cri-o://b431696cdbece4beff53510320a1491d5117615f770679c968ba920a50430a5b" gracePeriod=45
Nov 03 01:11:50.715307 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.715278    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 01:11:50.716228 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.716205    5012 generic.go:356] "Generic (PLEG): container finished" podID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerID="b431696cdbece4beff53510320a1491d5117615f770679c968ba920a50430a5b" exitCode=0
Nov 03 01:11:50.716343 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.716284    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" event={"ID":"170314c3-bfce-424b-bba2-3dc03d1b80e4","Type":"ContainerDied","Data":"b431696cdbece4beff53510320a1491d5117615f770679c968ba920a50430a5b"}
Nov 03 01:11:50.716343 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.716330    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt" event={"ID":"170314c3-bfce-424b-bba2-3dc03d1b80e4","Type":"ContainerDied","Data":"b4f2fb243d0a08eef24c64ee32fb84d31a1d9201728102c22477d71babc54203"}
Nov 03 01:11:50.716449 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.716345    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4f2fb243d0a08eef24c64ee32fb84d31a1d9201728102c22477d71babc54203"
Nov 03 01:11:50.717568 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.717550    5012 generic.go:356] "Generic (PLEG): container finished" podID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerID="c71c160d763be7b46f3e1c8cda3c3a14c6897d9c0fd32d7f10dbbcf47cc47d20" exitCode=0
Nov 03 01:11:50.717660 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.717584    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"
Nov 03 01:11:50.717660 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.717596    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" event={"ID":"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095","Type":"ContainerDied","Data":"c71c160d763be7b46f3e1c8cda3c3a14c6897d9c0fd32d7f10dbbcf47cc47d20"}
Nov 03 01:11:50.717660 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.717619    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px" event={"ID":"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095","Type":"ContainerDied","Data":"5620cfb64f1eba25935a4677d0b0314916551cbf39208c1abe493304eaad38e9"}
Nov 03 01:11:50.717660 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.717635    5012 scope.go:122] "RemoveContainer" containerID="c71c160d763be7b46f3e1c8cda3c3a14c6897d9c0fd32d7f10dbbcf47cc47d20"
Nov 03 01:11:50.718795 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.718775    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 01:11:50.727689 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.727666    5012 scope.go:122] "RemoveContainer" containerID="c71c160d763be7b46f3e1c8cda3c3a14c6897d9c0fd32d7f10dbbcf47cc47d20"
Nov 03 01:11:50.727936 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:11:50.727900    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c71c160d763be7b46f3e1c8cda3c3a14c6897d9c0fd32d7f10dbbcf47cc47d20\": container with ID starting with c71c160d763be7b46f3e1c8cda3c3a14c6897d9c0fd32d7f10dbbcf47cc47d20 not found: ID does not exist" containerID="c71c160d763be7b46f3e1c8cda3c3a14c6897d9c0fd32d7f10dbbcf47cc47d20"
Nov 03 01:11:50.727998 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.727944    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c71c160d763be7b46f3e1c8cda3c3a14c6897d9c0fd32d7f10dbbcf47cc47d20"} err="failed to get container status \"c71c160d763be7b46f3e1c8cda3c3a14c6897d9c0fd32d7f10dbbcf47cc47d20\": rpc error: code = NotFound desc = could not find container \"c71c160d763be7b46f3e1c8cda3c3a14c6897d9c0fd32d7f10dbbcf47cc47d20\": container with ID starting with c71c160d763be7b46f3e1c8cda3c3a14c6897d9c0fd32d7f10dbbcf47cc47d20 not found: ID does not exist"
Nov 03 01:11:50.765998 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.763360    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-dshm\") pod \"170314c3-bfce-424b-bba2-3dc03d1b80e4\" (UID: \"170314c3-bfce-424b-bba2-3dc03d1b80e4\") "
Nov 03 01:11:50.765998 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.763447    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-tmp-dir\") pod \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\" (UID: \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\") "
Nov 03 01:11:50.765998 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.763513    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-home\" (UniqueName: \"kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-home\") pod \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\" (UID: \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\") "
Nov 03 01:11:50.765998 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.763620    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/170314c3-bfce-424b-bba2-3dc03d1b80e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/170314c3-bfce-424b-bba2-3dc03d1b80e4-tls-certs\") pod \"170314c3-bfce-424b-bba2-3dc03d1b80e4\" (UID: \"170314c3-bfce-424b-bba2-3dc03d1b80e4\") "
Nov 03 01:11:50.765998 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.763671    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-model-cache\") pod \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\" (UID: \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\") "
Nov 03 01:11:50.765998 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.763705    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-model-cache\") pod \"170314c3-bfce-424b-bba2-3dc03d1b80e4\" (UID: \"170314c3-bfce-424b-bba2-3dc03d1b80e4\") "
Nov 03 01:11:50.765998 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.763755    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-dshm\") pod \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\" (UID: \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\") "
Nov 03 01:11:50.765998 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.763789    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-tmp-dir" pod "d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" (UID: "d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:50.765998 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.763791    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-tls-certs\" (UniqueName: \"kubernetes.io/secret/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-tls-certs\") pod \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\" (UID: \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\") "
Nov 03 01:11:50.765998 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.763865    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-tmp-dir\") pod \"170314c3-bfce-424b-bba2-3dc03d1b80e4\" (UID: \"170314c3-bfce-424b-bba2-3dc03d1b80e4\") "
Nov 03 01:11:50.765998 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.763945    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-home\" (UniqueName: \"kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-home\") pod \"170314c3-bfce-424b-bba2-3dc03d1b80e4\" (UID: \"170314c3-bfce-424b-bba2-3dc03d1b80e4\") "
Nov 03 01:11:50.765998 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.763997    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/170314c3-bfce-424b-bba2-3dc03d1b80e4-kube-api-access-b4gq9\" (UniqueName: \"kubernetes.io/projected/170314c3-bfce-424b-bba2-3dc03d1b80e4-kube-api-access-b4gq9\") pod \"170314c3-bfce-424b-bba2-3dc03d1b80e4\" (UID: \"170314c3-bfce-424b-bba2-3dc03d1b80e4\") "
Nov 03 01:11:50.765998 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.764064    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-kube-api-access-qjhvq\" (UniqueName: \"kubernetes.io/projected/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-kube-api-access-qjhvq\") pod \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\" (UID: \"d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095\") "
Nov 03 01:11:50.765998 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.764705    5012 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-tmp-dir\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:50.765998 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.764856    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-home" pod "170314c3-bfce-424b-bba2-3dc03d1b80e4" (UID: "170314c3-bfce-424b-bba2-3dc03d1b80e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:50.765998 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.765493    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-tmp-dir" pod "170314c3-bfce-424b-bba2-3dc03d1b80e4" (UID: "170314c3-bfce-424b-bba2-3dc03d1b80e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:50.765998 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.765534    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-home" pod "d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" (UID: "d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:50.765998 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.765830    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-model-cache" pod "170314c3-bfce-424b-bba2-3dc03d1b80e4" (UID: "170314c3-bfce-424b-bba2-3dc03d1b80e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:50.767922 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.766026    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-model-cache" pod "d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" (UID: "d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:50.767922 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.766194    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-dshm" pod "170314c3-bfce-424b-bba2-3dc03d1b80e4" (UID: "170314c3-bfce-424b-bba2-3dc03d1b80e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:50.768127 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.768099    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-tls-certs" pod "d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" (UID: "d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:11:50.773940 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.773870    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/170314c3-bfce-424b-bba2-3dc03d1b80e4-tls-certs" pod "170314c3-bfce-424b-bba2-3dc03d1b80e4" (UID: "170314c3-bfce-424b-bba2-3dc03d1b80e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:11:50.774037 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.773939    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-kube-api-access-qjhvq" pod "d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" (UID: "d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095"). InnerVolumeSpecName "kube-api-access-qjhvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:50.774037 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.773980    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/170314c3-bfce-424b-bba2-3dc03d1b80e4-kube-api-access-b4gq9" pod "170314c3-bfce-424b-bba2-3dc03d1b80e4" (UID: "170314c3-bfce-424b-bba2-3dc03d1b80e4"). InnerVolumeSpecName "kube-api-access-b4gq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:50.775511 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.775485    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-dshm" pod "d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" (UID: "d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:50.866069 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.866038    5012 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-dshm\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:50.866069 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.866066    5012 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-tls-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:50.866069 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.866077    5012 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-tmp-dir\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:50.866295 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.866085    5012 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-home\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:50.866295 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.866095    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4gq9\" (UniqueName: \"kubernetes.io/projected/170314c3-bfce-424b-bba2-3dc03d1b80e4-kube-api-access-b4gq9\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:50.866295 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.866104    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjhvq\" (UniqueName: \"kubernetes.io/projected/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-kube-api-access-qjhvq\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:50.866295 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.866113    5012 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-dshm\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:50.866295 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.866120    5012 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-home\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:50.866295 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.866128    5012 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/170314c3-bfce-424b-bba2-3dc03d1b80e4-tls-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:50.866295 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.866135    5012 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095-model-cache\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:50.866295 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:50.866143    5012 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/170314c3-bfce-424b-bba2-3dc03d1b80e4-model-cache\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:51.065141 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:51.065110    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"]
Nov 03 01:11:51.071132 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:51.071109    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/distinct-2-w3-8e90c5-kserve-575dcc5775-l84px"]
Nov 03 01:11:51.246649 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:51.246615    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" path="/var/lib/kubelet/pods/d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095/volumes"
Nov 03 01:11:51.560993 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:51.560954    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx" podUID="a314ceb2-941e-4c6d-8054-8c3bd7155692" containerName="payload-processing" containerID="cri-o://c60e8b81a090637a7da6b66c057b52c2e5854c4d522e6d1fb94fb47ae14bf7ed" gracePeriod=25
Nov 03 01:11:51.596636 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:51.596604    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8" podUID="fcb95377-3137-4676-b9b5-c464755a097f" containerName="payload-pre-processing" containerID="cri-o://487d2b26fca7ecc4f5d8bb3e90f30c7dfaa79995c9b983ece599f6747ba904f3" gracePeriod=25
Nov 03 01:11:51.727616 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:51.727578    5012 generic.go:356] "Generic (PLEG): container finished" podID="a314ceb2-941e-4c6d-8054-8c3bd7155692" containerID="c60e8b81a090637a7da6b66c057b52c2e5854c4d522e6d1fb94fb47ae14bf7ed" exitCode=0
Nov 03 01:11:51.727778 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:51.727618    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx" event={"ID":"a314ceb2-941e-4c6d-8054-8c3bd7155692","Type":"ContainerDied","Data":"c60e8b81a090637a7da6b66c057b52c2e5854c4d522e6d1fb94fb47ae14bf7ed"}
Nov 03 01:11:51.730889 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:51.730842    5012 generic.go:356] "Generic (PLEG): container finished" podID="fcb95377-3137-4676-b9b5-c464755a097f" containerID="487d2b26fca7ecc4f5d8bb3e90f30c7dfaa79995c9b983ece599f6747ba904f3" exitCode=0
Nov 03 01:11:51.730889 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:51.730880    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8" event={"ID":"fcb95377-3137-4676-b9b5-c464755a097f","Type":"ContainerDied","Data":"487d2b26fca7ecc4f5d8bb3e90f30c7dfaa79995c9b983ece599f6747ba904f3"}
Nov 03 01:11:51.731115 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:51.731075    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"
Nov 03 01:11:51.788695 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:51.788661    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"]
Nov 03 01:11:51.794686 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:51.794657    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/simulator-w3-8e90c5-kserve-5b47b97984-kbqwt"]
Nov 03 01:11:51.879832 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:51.879806    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx"
Nov 03 01:11:51.913450 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:51.913427    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8"
Nov 03 01:11:51.982266 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:51.982197    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a314ceb2-941e-4c6d-8054-8c3bd7155692-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a314ceb2-941e-4c6d-8054-8c3bd7155692-plugins-config-volume\") pod \"a314ceb2-941e-4c6d-8054-8c3bd7155692\" (UID: \"a314ceb2-941e-4c6d-8054-8c3bd7155692\") "
Nov 03 01:11:51.982266 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:51.982256    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcb95377-3137-4676-b9b5-c464755a097f-kube-api-access-sqt9z\" (UniqueName: \"kubernetes.io/projected/fcb95377-3137-4676-b9b5-c464755a097f-kube-api-access-sqt9z\") pod \"fcb95377-3137-4676-b9b5-c464755a097f\" (UID: \"fcb95377-3137-4676-b9b5-c464755a097f\") "
Nov 03 01:11:51.982508 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:51.982384    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcb95377-3137-4676-b9b5-c464755a097f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcb95377-3137-4676-b9b5-c464755a097f-plugins-config-volume\") pod \"fcb95377-3137-4676-b9b5-c464755a097f\" (UID: \"fcb95377-3137-4676-b9b5-c464755a097f\") "
Nov 03 01:11:51.982508 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:51.982431    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a314ceb2-941e-4c6d-8054-8c3bd7155692-kube-api-access-rwxqn\" (UniqueName: \"kubernetes.io/projected/a314ceb2-941e-4c6d-8054-8c3bd7155692-kube-api-access-rwxqn\") pod \"a314ceb2-941e-4c6d-8054-8c3bd7155692\" (UID: \"a314ceb2-941e-4c6d-8054-8c3bd7155692\") "
Nov 03 01:11:51.982657 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:51.982638    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a314ceb2-941e-4c6d-8054-8c3bd7155692-plugins-config-volume" pod "a314ceb2-941e-4c6d-8054-8c3bd7155692" (UID: "a314ceb2-941e-4c6d-8054-8c3bd7155692"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:11:51.982822 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:51.982795    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcb95377-3137-4676-b9b5-c464755a097f-plugins-config-volume" pod "fcb95377-3137-4676-b9b5-c464755a097f" (UID: "fcb95377-3137-4676-b9b5-c464755a097f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:11:51.983053 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:51.983030    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a314ceb2-941e-4c6d-8054-8c3bd7155692-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:51.983138 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:51.983058    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcb95377-3137-4676-b9b5-c464755a097f-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:51.984485 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:51.984460    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb95377-3137-4676-b9b5-c464755a097f-kube-api-access-sqt9z" pod "fcb95377-3137-4676-b9b5-c464755a097f" (UID: "fcb95377-3137-4676-b9b5-c464755a097f"). InnerVolumeSpecName "kube-api-access-sqt9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:51.984588 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:51.984533    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a314ceb2-941e-4c6d-8054-8c3bd7155692-kube-api-access-rwxqn" pod "a314ceb2-941e-4c6d-8054-8c3bd7155692" (UID: "a314ceb2-941e-4c6d-8054-8c3bd7155692"). InnerVolumeSpecName "kube-api-access-rwxqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:52.084440 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:52.084414    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqt9z\" (UniqueName: \"kubernetes.io/projected/fcb95377-3137-4676-b9b5-c464755a097f-kube-api-access-sqt9z\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:52.084440 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:52.084439    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwxqn\" (UniqueName: \"kubernetes.io/projected/a314ceb2-941e-4c6d-8054-8c3bd7155692-kube-api-access-rwxqn\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:52.739232 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:52.739165    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx"
Nov 03 01:11:52.739232 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:52.739212    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx" event={"ID":"a314ceb2-941e-4c6d-8054-8c3bd7155692","Type":"ContainerDied","Data":"b04d6c7f2e81a5402b2795c696ec6b513353c2df040c9cb30b3cd1872faa83bc"}
Nov 03 01:11:52.739829 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:52.739276    5012 scope.go:122] "RemoveContainer" containerID="c60e8b81a090637a7da6b66c057b52c2e5854c4d522e6d1fb94fb47ae14bf7ed"
Nov 03 01:11:52.741139 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:52.741112    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8" event={"ID":"fcb95377-3137-4676-b9b5-c464755a097f","Type":"ContainerDied","Data":"10d35c58a161230f12503919dfa7f5be9fb3c5151f0ee3b334c6498929389a22"}
Nov 03 01:11:52.741301 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:52.741197    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8"
Nov 03 01:11:52.753040 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:52.753016    5012 scope.go:122] "RemoveContainer" containerID="487d2b26fca7ecc4f5d8bb3e90f30c7dfaa79995c9b983ece599f6747ba904f3"
Nov 03 01:11:52.802128 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:52.802103    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8"]
Nov 03 01:11:52.807528 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:52.807502    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8e90c5-865b959567-xjrj8"]
Nov 03 01:11:52.837476 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:52.837453    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx"]
Nov 03 01:11:52.841343 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:52.841320    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8e90c5-75f87f7845-bbfrx"]
Nov 03 01:11:53.244619 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:53.244583    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" path="/var/lib/kubelet/pods/170314c3-bfce-424b-bba2-3dc03d1b80e4/volumes"
Nov 03 01:11:53.245027 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:53.245011    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a314ceb2-941e-4c6d-8054-8c3bd7155692" path="/var/lib/kubelet/pods/a314ceb2-941e-4c6d-8054-8c3bd7155692/volumes"
Nov 03 01:11:53.245408 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:53.245394    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcb95377-3137-4676-b9b5-c464755a097f" path="/var/lib/kubelet/pods/fcb95377-3137-4676-b9b5-c464755a097f/volumes"
Nov 03 01:11:54.749633 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:54.749598    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd" podUID="5d0c60ba-1ab0-4a70-82b9-78580583e5cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:11:56.958340 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:56.958317    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd"
Nov 03 01:11:57.067642 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:57.067608    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0c60ba-1ab0-4a70-82b9-78580583e5cf-kube-api-access-w56g4\" (UniqueName: \"kubernetes.io/projected/5d0c60ba-1ab0-4a70-82b9-78580583e5cf-kube-api-access-w56g4\") pod \"5d0c60ba-1ab0-4a70-82b9-78580583e5cf\" (UID: \"5d0c60ba-1ab0-4a70-82b9-78580583e5cf\") "
Nov 03 01:11:57.067838 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:57.067752    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d0c60ba-1ab0-4a70-82b9-78580583e5cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d0c60ba-1ab0-4a70-82b9-78580583e5cf-plugins-config-volume\") pod \"5d0c60ba-1ab0-4a70-82b9-78580583e5cf\" (UID: \"5d0c60ba-1ab0-4a70-82b9-78580583e5cf\") "
Nov 03 01:11:57.068161 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:57.068136    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d0c60ba-1ab0-4a70-82b9-78580583e5cf-plugins-config-volume" pod "5d0c60ba-1ab0-4a70-82b9-78580583e5cf" (UID: "5d0c60ba-1ab0-4a70-82b9-78580583e5cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:11:57.069804 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:57.069773    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0c60ba-1ab0-4a70-82b9-78580583e5cf-kube-api-access-w56g4" pod "5d0c60ba-1ab0-4a70-82b9-78580583e5cf" (UID: "5d0c60ba-1ab0-4a70-82b9-78580583e5cf"). InnerVolumeSpecName "kube-api-access-w56g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:57.169776 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:57.169747    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w56g4\" (UniqueName: \"kubernetes.io/projected/5d0c60ba-1ab0-4a70-82b9-78580583e5cf-kube-api-access-w56g4\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:57.169776 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:57.169775    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d0c60ba-1ab0-4a70-82b9-78580583e5cf-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:11:57.782229 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:57.782132    5012 generic.go:356] "Generic (PLEG): container finished" podID="5d0c60ba-1ab0-4a70-82b9-78580583e5cf" containerID="119c0b800c3fa39ed88d4b22b3a430413422c2eb25665093cd221ef745d67a37" exitCode=0
Nov 03 01:11:57.782229 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:57.782190    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd"
Nov 03 01:11:57.782229 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:57.782192    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd" event={"ID":"5d0c60ba-1ab0-4a70-82b9-78580583e5cf","Type":"ContainerDied","Data":"119c0b800c3fa39ed88d4b22b3a430413422c2eb25665093cd221ef745d67a37"}
Nov 03 01:11:57.782557 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:57.782256    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd" event={"ID":"5d0c60ba-1ab0-4a70-82b9-78580583e5cf","Type":"ContainerDied","Data":"18dd9584c6017173522ae4764a00b3ff8eef53db3aa1101cb71549a0ea09f08f"}
Nov 03 01:11:57.782557 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:57.782289    5012 scope.go:122] "RemoveContainer" containerID="119c0b800c3fa39ed88d4b22b3a430413422c2eb25665093cd221ef745d67a37"
Nov 03 01:11:57.793840 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:57.793601    5012 scope.go:122] "RemoveContainer" containerID="119c0b800c3fa39ed88d4b22b3a430413422c2eb25665093cd221ef745d67a37"
Nov 03 01:11:57.793961 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:11:57.793940    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"119c0b800c3fa39ed88d4b22b3a430413422c2eb25665093cd221ef745d67a37\": container with ID starting with 119c0b800c3fa39ed88d4b22b3a430413422c2eb25665093cd221ef745d67a37 not found: ID does not exist" containerID="119c0b800c3fa39ed88d4b22b3a430413422c2eb25665093cd221ef745d67a37"
Nov 03 01:11:57.794017 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:57.793969    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"119c0b800c3fa39ed88d4b22b3a430413422c2eb25665093cd221ef745d67a37"} err="failed to get container status \"119c0b800c3fa39ed88d4b22b3a430413422c2eb25665093cd221ef745d67a37\": rpc error: code = NotFound desc = could not find container \"119c0b800c3fa39ed88d4b22b3a430413422c2eb25665093cd221ef745d67a37\": container with ID starting with 119c0b800c3fa39ed88d4b22b3a430413422c2eb25665093cd221ef745d67a37 not found: ID does not exist"
Nov 03 01:11:57.830684 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:57.830654    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd"]
Nov 03 01:11:57.840863 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:57.840839    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e62935-5689cd54bf-g9tmd"]
Nov 03 01:11:59.244540 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:11:59.244508    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d0c60ba-1ab0-4a70-82b9-78580583e5cf" path="/var/lib/kubelet/pods/5d0c60ba-1ab0-4a70-82b9-78580583e5cf/volumes"
Nov 03 01:12:04.589362 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:04.589326    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"]
Nov 03 01:12:04.589836 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:04.589569    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94" podUID="5795e973-761d-43af-91ea-0451024c0d42" containerName="istio-proxy" containerID="cri-o://fee4d8bff4b21c9b375b12109683697675d5a69deda156a04e6d759b3c54faab" gracePeriod=30
Nov 03 01:12:04.954134 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:04.954098    5012 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:12:04.954308 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:04.954175    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94" podUID="5795e973-761d-43af-91ea-0451024c0d42" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:12:07.362606 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:12:07.362567    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:12:09.878604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:09.878568    5012 generic.go:356] "Generic (PLEG): container finished" podID="5795e973-761d-43af-91ea-0451024c0d42" containerID="fee4d8bff4b21c9b375b12109683697675d5a69deda156a04e6d759b3c54faab" exitCode=0
Nov 03 01:12:09.879047 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:09.878634    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94" event={"ID":"5795e973-761d-43af-91ea-0451024c0d42","Type":"ContainerDied","Data":"fee4d8bff4b21c9b375b12109683697675d5a69deda156a04e6d759b3c54faab"}
Nov 03 01:12:09.879047 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:09.878697    5012 scope.go:122] "RemoveContainer" containerID="793b72e6c7c6c379be340ef4e6637111ace89e4ca35f2443ed4fcaf265e01488"
Nov 03 01:12:09.903900 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:09.903877    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 01:12:10.049103 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.049070    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5795e973-761d-43af-91ea-0451024c0d42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5795e973-761d-43af-91ea-0451024c0d42-istio-podinfo\") pod \"5795e973-761d-43af-91ea-0451024c0d42\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") "
Nov 03 01:12:10.049244 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.049116    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-credential-socket\") pod \"5795e973-761d-43af-91ea-0451024c0d42\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") "
Nov 03 01:12:10.049244 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.049148    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5795e973-761d-43af-91ea-0451024c0d42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5795e973-761d-43af-91ea-0451024c0d42-istiod-ca-cert\") pod \"5795e973-761d-43af-91ea-0451024c0d42\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") "
Nov 03 01:12:10.049244 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.049190    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5795e973-761d-43af-91ea-0451024c0d42-istio-token\" (UniqueName: \"kubernetes.io/projected/5795e973-761d-43af-91ea-0451024c0d42-istio-token\") pod \"5795e973-761d-43af-91ea-0451024c0d42\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") "
Nov 03 01:12:10.049244 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.049223    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-istio-envoy\") pod \"5795e973-761d-43af-91ea-0451024c0d42\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") "
Nov 03 01:12:10.049466 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.049282    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-workload-certs\") pod \"5795e973-761d-43af-91ea-0451024c0d42\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") "
Nov 03 01:12:10.049466 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.049311    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5795e973-761d-43af-91ea-0451024c0d42-kube-api-access-82mgl\" (UniqueName: \"kubernetes.io/projected/5795e973-761d-43af-91ea-0451024c0d42-kube-api-access-82mgl\") pod \"5795e973-761d-43af-91ea-0451024c0d42\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") "
Nov 03 01:12:10.049466 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.049412    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-workload-socket\") pod \"5795e973-761d-43af-91ea-0451024c0d42\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") "
Nov 03 01:12:10.049616 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.049454    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-credential-socket" pod "5795e973-761d-43af-91ea-0451024c0d42" (UID: "5795e973-761d-43af-91ea-0451024c0d42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:12:10.049616 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.049499    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-istio-data\") pod \"5795e973-761d-43af-91ea-0451024c0d42\" (UID: \"5795e973-761d-43af-91ea-0451024c0d42\") "
Nov 03 01:12:10.049722 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.049642    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-workload-certs" pod "5795e973-761d-43af-91ea-0451024c0d42" (UID: "5795e973-761d-43af-91ea-0451024c0d42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:12:10.049781 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.049739    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-workload-socket" pod "5795e973-761d-43af-91ea-0451024c0d42" (UID: "5795e973-761d-43af-91ea-0451024c0d42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:12:10.049781 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.049758    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5795e973-761d-43af-91ea-0451024c0d42-istiod-ca-cert" pod "5795e973-761d-43af-91ea-0451024c0d42" (UID: "5795e973-761d-43af-91ea-0451024c0d42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:12:10.049974 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.049954    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-istio-data" pod "5795e973-761d-43af-91ea-0451024c0d42" (UID: "5795e973-761d-43af-91ea-0451024c0d42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:12:10.050099 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.050082    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:12:10.050169 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.050101    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:12:10.050169 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.050111    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5795e973-761d-43af-91ea-0451024c0d42-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:12:10.050169 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.050119    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:12:10.050169 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.050128    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:12:10.051435 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.051411    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5795e973-761d-43af-91ea-0451024c0d42-istio-podinfo" pod "5795e973-761d-43af-91ea-0451024c0d42" (UID: "5795e973-761d-43af-91ea-0451024c0d42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:12:10.051532 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.051487    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-istio-envoy" pod "5795e973-761d-43af-91ea-0451024c0d42" (UID: "5795e973-761d-43af-91ea-0451024c0d42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:12:10.051901 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.051884    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5795e973-761d-43af-91ea-0451024c0d42-kube-api-access-82mgl" pod "5795e973-761d-43af-91ea-0451024c0d42" (UID: "5795e973-761d-43af-91ea-0451024c0d42"). InnerVolumeSpecName "kube-api-access-82mgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:12:10.051973 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.051945    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5795e973-761d-43af-91ea-0451024c0d42-istio-token" pod "5795e973-761d-43af-91ea-0451024c0d42" (UID: "5795e973-761d-43af-91ea-0451024c0d42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:12:10.151696 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.151644    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5795e973-761d-43af-91ea-0451024c0d42-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:12:10.151696 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.151666    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5795e973-761d-43af-91ea-0451024c0d42-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:12:10.151696 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.151676    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5795e973-761d-43af-91ea-0451024c0d42-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:12:10.151696 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.151684    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82mgl\" (UniqueName: \"kubernetes.io/projected/5795e973-761d-43af-91ea-0451024c0d42-kube-api-access-82mgl\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:12:10.887707 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.887673    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"
Nov 03 01:12:10.888213 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.887719    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94" event={"ID":"5795e973-761d-43af-91ea-0451024c0d42","Type":"ContainerDied","Data":"b099b0c658cfd5a6e6491edf03d71bae61814ef61a8d9696334b9166b1d3d2fe"}
Nov 03 01:12:10.888213 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.887779    5012 scope.go:122] "RemoveContainer" containerID="fee4d8bff4b21c9b375b12109683697675d5a69deda156a04e6d759b3c54faab"
Nov 03 01:12:10.984022 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.983993    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"]
Nov 03 01:12:10.990946 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:10.990919    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-2zp94"]
Nov 03 01:12:11.245202 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:11.245126    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5795e973-761d-43af-91ea-0451024c0d42" path="/var/lib/kubelet/pods/5795e973-761d-43af-91ea-0451024c0d42/volumes"
Nov 03 01:12:21.588025 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:21.587986    5012 scope.go:122] "RemoveContainer" containerID="b431696cdbece4beff53510320a1491d5117615f770679c968ba920a50430a5b"
Nov 03 01:12:35.922813 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:35.922769    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w"]
Nov 03 01:12:35.923504 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:35.923027    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w" podUID="e9bafb53-588d-4773-8c0b-aa8682c19d37" containerName="maas-api" containerID="cri-o://a09ff37d360db30848e163617faf4d0c57755d1b8d082c15530af66f9a0dcfbe" gracePeriod=30
Nov 03 01:12:36.142663 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:36.142627    5012 generic.go:356] "Generic (PLEG): container finished" podID="e9bafb53-588d-4773-8c0b-aa8682c19d37" containerID="a09ff37d360db30848e163617faf4d0c57755d1b8d082c15530af66f9a0dcfbe" exitCode=0
Nov 03 01:12:36.142840 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:36.142697    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w" event={"ID":"e9bafb53-588d-4773-8c0b-aa8682c19d37","Type":"ContainerDied","Data":"a09ff37d360db30848e163617faf4d0c57755d1b8d082c15530af66f9a0dcfbe"}
Nov 03 01:12:36.188562 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:36.188496    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w"
Nov 03 01:12:36.301477 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:36.301437    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9bafb53-588d-4773-8c0b-aa8682c19d37-kube-api-access-cl8kf\" (UniqueName: \"kubernetes.io/projected/e9bafb53-588d-4773-8c0b-aa8682c19d37-kube-api-access-cl8kf\") pod \"e9bafb53-588d-4773-8c0b-aa8682c19d37\" (UID: \"e9bafb53-588d-4773-8c0b-aa8682c19d37\") "
Nov 03 01:12:36.301669 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:36.301525    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9bafb53-588d-4773-8c0b-aa8682c19d37-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9bafb53-588d-4773-8c0b-aa8682c19d37-metrics-certs\") pod \"e9bafb53-588d-4773-8c0b-aa8682c19d37\" (UID: \"e9bafb53-588d-4773-8c0b-aa8682c19d37\") "
Nov 03 01:12:36.301727 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:36.301710    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9bafb53-588d-4773-8c0b-aa8682c19d37-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9bafb53-588d-4773-8c0b-aa8682c19d37-openshift-service-ca\") pod \"e9bafb53-588d-4773-8c0b-aa8682c19d37\" (UID: \"e9bafb53-588d-4773-8c0b-aa8682c19d37\") "
Nov 03 01:12:36.301861 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:36.301844    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9bafb53-588d-4773-8c0b-aa8682c19d37-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9bafb53-588d-4773-8c0b-aa8682c19d37-maas-api-tls\") pod \"e9bafb53-588d-4773-8c0b-aa8682c19d37\" (UID: \"e9bafb53-588d-4773-8c0b-aa8682c19d37\") "
Nov 03 01:12:36.302188 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:36.302157    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9bafb53-588d-4773-8c0b-aa8682c19d37-openshift-service-ca" pod "e9bafb53-588d-4773-8c0b-aa8682c19d37" (UID: "e9bafb53-588d-4773-8c0b-aa8682c19d37"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:12:36.302442 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:36.302412    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9bafb53-588d-4773-8c0b-aa8682c19d37-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:12:36.304289 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:36.304265    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9bafb53-588d-4773-8c0b-aa8682c19d37-maas-api-tls" pod "e9bafb53-588d-4773-8c0b-aa8682c19d37" (UID: "e9bafb53-588d-4773-8c0b-aa8682c19d37"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:12:36.304924 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:36.304886    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9bafb53-588d-4773-8c0b-aa8682c19d37-kube-api-access-cl8kf" pod "e9bafb53-588d-4773-8c0b-aa8682c19d37" (UID: "e9bafb53-588d-4773-8c0b-aa8682c19d37"). InnerVolumeSpecName "kube-api-access-cl8kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:12:36.305033 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:36.304890    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9bafb53-588d-4773-8c0b-aa8682c19d37-metrics-certs" pod "e9bafb53-588d-4773-8c0b-aa8682c19d37" (UID: "e9bafb53-588d-4773-8c0b-aa8682c19d37"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:12:36.406932 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:36.404925    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9bafb53-588d-4773-8c0b-aa8682c19d37-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:12:36.406932 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:36.404963    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9bafb53-588d-4773-8c0b-aa8682c19d37-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:12:36.406932 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:36.404978    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl8kf\" (UniqueName: \"kubernetes.io/projected/e9bafb53-588d-4773-8c0b-aa8682c19d37-kube-api-access-cl8kf\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:12:37.151263 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:37.151228    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w"
Nov 03 01:12:37.151791 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:37.151259    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w" event={"ID":"e9bafb53-588d-4773-8c0b-aa8682c19d37","Type":"ContainerDied","Data":"749b86cb0bbc85ca67f1131b589ec6c5942ee39b39d6ff2230a1680970f2a8fa"}
Nov 03 01:12:37.151791 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:37.151320    5012 scope.go:122] "RemoveContainer" containerID="a09ff37d360db30848e163617faf4d0c57755d1b8d082c15530af66f9a0dcfbe"
Nov 03 01:12:37.205350 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:37.205327    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w"]
Nov 03 01:12:37.215376 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:37.215354    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a07a90-6744794f7f-9929w"]
Nov 03 01:12:37.245357 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:37.245331    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9bafb53-588d-4773-8c0b-aa8682c19d37" path="/var/lib/kubelet/pods/e9bafb53-588d-4773-8c0b-aa8682c19d37/volumes"
Nov 03 01:12:37.363077 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:12:37.363047    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:12:52.723213 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:52.723173    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5"]
Nov 03 01:12:52.754123 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:52.754085    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl"]
Nov 03 01:12:57.728969 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:57.727982    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5" podUID="cafedf4f-719f-4e62-9f27-28f2e2017131" containerName="payload-processing" containerID="cri-o://4c379a996cbbb27f2e418e4ce0af3d2e442b6c125e65d2f345f8eec52184d1ff" gracePeriod=25
Nov 03 01:12:57.754938 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:12:57.754636    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl" podUID="d4aa636c-5490-4e79-9b11-3284af3e64a6" containerName="payload-pre-processing" containerID="cri-o://88ac4c55a2000b4ce1146da1ab9a314af0e2e3d0317c7cc81358a71873f3d91d" gracePeriod=25
Nov 03 01:13:00.362945 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:00.362888    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5" podUID="cafedf4f-719f-4e62-9f27-28f2e2017131" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:13:00.363457 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:00.362889    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl" podUID="d4aa636c-5490-4e79-9b11-3284af3e64a6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:13:04.554171 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:04.554133    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb"]
Nov 03 01:13:04.583805 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:04.583768    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw"]
Nov 03 01:13:07.364673 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:13:07.364638    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:13:09.554952 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:09.554888    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb" podUID="373c74f5-bc8f-4938-b64f-f4ab8d6bb993" containerName="payload-processing" containerID="cri-o://71a8b09dd01a0eba38aecd97932469c88015e2b204465eaae74b791e78085709" gracePeriod=25
Nov 03 01:13:09.585011 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:09.584965    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw" podUID="bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4" containerName="payload-pre-processing" containerID="cri-o://627544b5c7d32d09faf4b80b0fd860d862727c1461723655ee351598b2e01f01" gracePeriod=25
Nov 03 01:13:09.880963 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:09.880922    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb" podUID="373c74f5-bc8f-4938-b64f-f4ab8d6bb993" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:13:09.880963 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:09.880940    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw" podUID="bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:13:10.363716 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:10.363680    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5" podUID="cafedf4f-719f-4e62-9f27-28f2e2017131" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:13:10.363937 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:10.363744    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl" podUID="d4aa636c-5490-4e79-9b11-3284af3e64a6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:13:15.663512 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:15.663475    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft"]
Nov 03 01:13:15.664211 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:15.663740    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft" podUID="5cda386d-6a1d-4fd9-95c5-1a68750e10b0" containerName="maas-api" containerID="cri-o://d27aebf916c104eac2ffa546ede69c2ba0ae12b95fc4308346c4e4db5bca6653" gracePeriod=30
Nov 03 01:13:15.991143 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:15.991110    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft"
Nov 03 01:13:16.140465 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:16.140432    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-openshift-service-ca\") pod \"5cda386d-6a1d-4fd9-95c5-1a68750e10b0\" (UID: \"5cda386d-6a1d-4fd9-95c5-1a68750e10b0\") "
Nov 03 01:13:16.140668 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:16.140532    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-maas-api-tls\") pod \"5cda386d-6a1d-4fd9-95c5-1a68750e10b0\" (UID: \"5cda386d-6a1d-4fd9-95c5-1a68750e10b0\") "
Nov 03 01:13:16.140668 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:16.140592    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-metrics-certs\") pod \"5cda386d-6a1d-4fd9-95c5-1a68750e10b0\" (UID: \"5cda386d-6a1d-4fd9-95c5-1a68750e10b0\") "
Nov 03 01:13:16.140798 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:16.140692    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-kube-api-access-tg9dv\" (UniqueName: \"kubernetes.io/projected/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-kube-api-access-tg9dv\") pod \"5cda386d-6a1d-4fd9-95c5-1a68750e10b0\" (UID: \"5cda386d-6a1d-4fd9-95c5-1a68750e10b0\") "
Nov 03 01:13:16.140888 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:16.140859    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-openshift-service-ca" pod "5cda386d-6a1d-4fd9-95c5-1a68750e10b0" (UID: "5cda386d-6a1d-4fd9-95c5-1a68750e10b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:13:16.141227 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:16.141201    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:13:16.143782 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:16.143754    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-metrics-certs" pod "5cda386d-6a1d-4fd9-95c5-1a68750e10b0" (UID: "5cda386d-6a1d-4fd9-95c5-1a68750e10b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:13:16.143782 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:16.143766    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-kube-api-access-tg9dv" pod "5cda386d-6a1d-4fd9-95c5-1a68750e10b0" (UID: "5cda386d-6a1d-4fd9-95c5-1a68750e10b0"). InnerVolumeSpecName "kube-api-access-tg9dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:13:16.146213 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:16.146181    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-maas-api-tls" pod "5cda386d-6a1d-4fd9-95c5-1a68750e10b0" (UID: "5cda386d-6a1d-4fd9-95c5-1a68750e10b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:13:16.263438 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:16.252234    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:13:16.263438 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:16.252273    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:13:16.263438 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:16.252289    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg9dv\" (UniqueName: \"kubernetes.io/projected/5cda386d-6a1d-4fd9-95c5-1a68750e10b0-kube-api-access-tg9dv\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:13:16.554271 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:16.554192    5012 generic.go:356] "Generic (PLEG): container finished" podID="5cda386d-6a1d-4fd9-95c5-1a68750e10b0" containerID="d27aebf916c104eac2ffa546ede69c2ba0ae12b95fc4308346c4e4db5bca6653" exitCode=0
Nov 03 01:13:16.554271 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:16.554250    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft"
Nov 03 01:13:16.554455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:16.554272    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft" event={"ID":"5cda386d-6a1d-4fd9-95c5-1a68750e10b0","Type":"ContainerDied","Data":"d27aebf916c104eac2ffa546ede69c2ba0ae12b95fc4308346c4e4db5bca6653"}
Nov 03 01:13:16.554455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:16.554347    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft" event={"ID":"5cda386d-6a1d-4fd9-95c5-1a68750e10b0","Type":"ContainerDied","Data":"02d6670cc77dbfe50a11e5a181a3a7a938f142631ba0eb3c1e867a52bd37ff4f"}
Nov 03 01:13:16.554455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:16.554369    5012 scope.go:122] "RemoveContainer" containerID="d27aebf916c104eac2ffa546ede69c2ba0ae12b95fc4308346c4e4db5bca6653"
Nov 03 01:13:16.565077 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:16.565060    5012 scope.go:122] "RemoveContainer" containerID="d27aebf916c104eac2ffa546ede69c2ba0ae12b95fc4308346c4e4db5bca6653"
Nov 03 01:13:16.565371 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:13:16.565350    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d27aebf916c104eac2ffa546ede69c2ba0ae12b95fc4308346c4e4db5bca6653\": container with ID starting with d27aebf916c104eac2ffa546ede69c2ba0ae12b95fc4308346c4e4db5bca6653 not found: ID does not exist" containerID="d27aebf916c104eac2ffa546ede69c2ba0ae12b95fc4308346c4e4db5bca6653"
Nov 03 01:13:16.565439 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:16.565379    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d27aebf916c104eac2ffa546ede69c2ba0ae12b95fc4308346c4e4db5bca6653"} err="failed to get container status \"d27aebf916c104eac2ffa546ede69c2ba0ae12b95fc4308346c4e4db5bca6653\": rpc error: code = NotFound desc = could not find container \"d27aebf916c104eac2ffa546ede69c2ba0ae12b95fc4308346c4e4db5bca6653\": container with ID starting with d27aebf916c104eac2ffa546ede69c2ba0ae12b95fc4308346c4e4db5bca6653 not found: ID does not exist"
Nov 03 01:13:16.603736 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:16.603709    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft"]
Nov 03 01:13:16.614160 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:16.614136    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d9e067-8684bcdd87-bh2ft"]
Nov 03 01:13:17.245387 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:17.245355    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cda386d-6a1d-4fd9-95c5-1a68750e10b0" path="/var/lib/kubelet/pods/5cda386d-6a1d-4fd9-95c5-1a68750e10b0/volumes"
Nov 03 01:13:18.044507 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.044477    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5"
Nov 03 01:13:18.066904 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.066878    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl"
Nov 03 01:13:18.186964 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.186872    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cafedf4f-719f-4e62-9f27-28f2e2017131-kube-api-access-5w8ml\" (UniqueName: \"kubernetes.io/projected/cafedf4f-719f-4e62-9f27-28f2e2017131-kube-api-access-5w8ml\") pod \"cafedf4f-719f-4e62-9f27-28f2e2017131\" (UID: \"cafedf4f-719f-4e62-9f27-28f2e2017131\") "
Nov 03 01:13:18.187171 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.187115    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4aa636c-5490-4e79-9b11-3284af3e64a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4aa636c-5490-4e79-9b11-3284af3e64a6-plugins-config-volume\") pod \"d4aa636c-5490-4e79-9b11-3284af3e64a6\" (UID: \"d4aa636c-5490-4e79-9b11-3284af3e64a6\") "
Nov 03 01:13:18.187289 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.187273    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4aa636c-5490-4e79-9b11-3284af3e64a6-kube-api-access-mmv4g\" (UniqueName: \"kubernetes.io/projected/d4aa636c-5490-4e79-9b11-3284af3e64a6-kube-api-access-mmv4g\") pod \"d4aa636c-5490-4e79-9b11-3284af3e64a6\" (UID: \"d4aa636c-5490-4e79-9b11-3284af3e64a6\") "
Nov 03 01:13:18.187361 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.187321    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cafedf4f-719f-4e62-9f27-28f2e2017131-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cafedf4f-719f-4e62-9f27-28f2e2017131-plugins-config-volume\") pod \"cafedf4f-719f-4e62-9f27-28f2e2017131\" (UID: \"cafedf4f-719f-4e62-9f27-28f2e2017131\") "
Nov 03 01:13:18.188162 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.188131    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4aa636c-5490-4e79-9b11-3284af3e64a6-plugins-config-volume" pod "d4aa636c-5490-4e79-9b11-3284af3e64a6" (UID: "d4aa636c-5490-4e79-9b11-3284af3e64a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:13:18.188692 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.188659    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cafedf4f-719f-4e62-9f27-28f2e2017131-plugins-config-volume" pod "cafedf4f-719f-4e62-9f27-28f2e2017131" (UID: "cafedf4f-719f-4e62-9f27-28f2e2017131"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:13:18.211932 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.205442    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafedf4f-719f-4e62-9f27-28f2e2017131-kube-api-access-5w8ml" pod "cafedf4f-719f-4e62-9f27-28f2e2017131" (UID: "cafedf4f-719f-4e62-9f27-28f2e2017131"). InnerVolumeSpecName "kube-api-access-5w8ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:13:18.219582 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.215115    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4aa636c-5490-4e79-9b11-3284af3e64a6-kube-api-access-mmv4g" pod "d4aa636c-5490-4e79-9b11-3284af3e64a6" (UID: "d4aa636c-5490-4e79-9b11-3284af3e64a6"). InnerVolumeSpecName "kube-api-access-mmv4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:13:18.288551 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.288473    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmv4g\" (UniqueName: \"kubernetes.io/projected/d4aa636c-5490-4e79-9b11-3284af3e64a6-kube-api-access-mmv4g\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:13:18.288551 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.288502    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cafedf4f-719f-4e62-9f27-28f2e2017131-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:13:18.288551 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.288511    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w8ml\" (UniqueName: \"kubernetes.io/projected/cafedf4f-719f-4e62-9f27-28f2e2017131-kube-api-access-5w8ml\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:13:18.288551 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.288521    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4aa636c-5490-4e79-9b11-3284af3e64a6-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:13:18.571044 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.570955    5012 generic.go:356] "Generic (PLEG): container finished" podID="d4aa636c-5490-4e79-9b11-3284af3e64a6" containerID="88ac4c55a2000b4ce1146da1ab9a314af0e2e3d0317c7cc81358a71873f3d91d" exitCode=0
Nov 03 01:13:18.571044 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.571011    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl"
Nov 03 01:13:18.571291 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.571037    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl" event={"ID":"d4aa636c-5490-4e79-9b11-3284af3e64a6","Type":"ContainerDied","Data":"88ac4c55a2000b4ce1146da1ab9a314af0e2e3d0317c7cc81358a71873f3d91d"}
Nov 03 01:13:18.571291 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.571095    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl" event={"ID":"d4aa636c-5490-4e79-9b11-3284af3e64a6","Type":"ContainerDied","Data":"38c7548ad003d582c500fe1a996f005c8d428dcdae44cc7b754506d64bf8104e"}
Nov 03 01:13:18.571291 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.571122    5012 scope.go:122] "RemoveContainer" containerID="88ac4c55a2000b4ce1146da1ab9a314af0e2e3d0317c7cc81358a71873f3d91d"
Nov 03 01:13:18.572700 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.572673    5012 generic.go:356] "Generic (PLEG): container finished" podID="cafedf4f-719f-4e62-9f27-28f2e2017131" containerID="4c379a996cbbb27f2e418e4ce0af3d2e442b6c125e65d2f345f8eec52184d1ff" exitCode=0
Nov 03 01:13:18.572820 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.572716    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5"
Nov 03 01:13:18.572820 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.572724    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5" event={"ID":"cafedf4f-719f-4e62-9f27-28f2e2017131","Type":"ContainerDied","Data":"4c379a996cbbb27f2e418e4ce0af3d2e442b6c125e65d2f345f8eec52184d1ff"}
Nov 03 01:13:18.572820 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.572777    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5" event={"ID":"cafedf4f-719f-4e62-9f27-28f2e2017131","Type":"ContainerDied","Data":"d53acb0d60a769c4f0fbbf2da01933d62f4b38cd347e953bee0a24de8776e99f"}
Nov 03 01:13:18.582166 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.582146    5012 scope.go:122] "RemoveContainer" containerID="88ac4c55a2000b4ce1146da1ab9a314af0e2e3d0317c7cc81358a71873f3d91d"
Nov 03 01:13:18.582508 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:13:18.582418    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ac4c55a2000b4ce1146da1ab9a314af0e2e3d0317c7cc81358a71873f3d91d\": container with ID starting with 88ac4c55a2000b4ce1146da1ab9a314af0e2e3d0317c7cc81358a71873f3d91d not found: ID does not exist" containerID="88ac4c55a2000b4ce1146da1ab9a314af0e2e3d0317c7cc81358a71873f3d91d"
Nov 03 01:13:18.582508 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.582456    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ac4c55a2000b4ce1146da1ab9a314af0e2e3d0317c7cc81358a71873f3d91d"} err="failed to get container status \"88ac4c55a2000b4ce1146da1ab9a314af0e2e3d0317c7cc81358a71873f3d91d\": rpc error: code = NotFound desc = could not find container \"88ac4c55a2000b4ce1146da1ab9a314af0e2e3d0317c7cc81358a71873f3d91d\": container with ID starting with 88ac4c55a2000b4ce1146da1ab9a314af0e2e3d0317c7cc81358a71873f3d91d not found: ID does not exist"
Nov 03 01:13:18.582508 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.582481    5012 scope.go:122] "RemoveContainer" containerID="4c379a996cbbb27f2e418e4ce0af3d2e442b6c125e65d2f345f8eec52184d1ff"
Nov 03 01:13:18.591759 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.591743    5012 scope.go:122] "RemoveContainer" containerID="4c379a996cbbb27f2e418e4ce0af3d2e442b6c125e65d2f345f8eec52184d1ff"
Nov 03 01:13:18.592008 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:13:18.591991    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c379a996cbbb27f2e418e4ce0af3d2e442b6c125e65d2f345f8eec52184d1ff\": container with ID starting with 4c379a996cbbb27f2e418e4ce0af3d2e442b6c125e65d2f345f8eec52184d1ff not found: ID does not exist" containerID="4c379a996cbbb27f2e418e4ce0af3d2e442b6c125e65d2f345f8eec52184d1ff"
Nov 03 01:13:18.592062 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.592014    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c379a996cbbb27f2e418e4ce0af3d2e442b6c125e65d2f345f8eec52184d1ff"} err="failed to get container status \"4c379a996cbbb27f2e418e4ce0af3d2e442b6c125e65d2f345f8eec52184d1ff\": rpc error: code = NotFound desc = could not find container \"4c379a996cbbb27f2e418e4ce0af3d2e442b6c125e65d2f345f8eec52184d1ff\": container with ID starting with 4c379a996cbbb27f2e418e4ce0af3d2e442b6c125e65d2f345f8eec52184d1ff not found: ID does not exist"
Nov 03 01:13:18.617243 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.617220    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5"]
Nov 03 01:13:18.627465 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.627447    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d9e067-65c5df4dd9-5vfl5"]
Nov 03 01:13:18.656167 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.656147    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl"]
Nov 03 01:13:18.659737 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:18.659706    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d9e067-5dfc99456c-nf5kl"]
Nov 03 01:13:19.244994 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:19.244962    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cafedf4f-719f-4e62-9f27-28f2e2017131" path="/var/lib/kubelet/pods/cafedf4f-719f-4e62-9f27-28f2e2017131/volumes"
Nov 03 01:13:19.245386 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:19.245368    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4aa636c-5490-4e79-9b11-3284af3e64a6" path="/var/lib/kubelet/pods/d4aa636c-5490-4e79-9b11-3284af3e64a6/volumes"
Nov 03 01:13:19.881094 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:19.881056    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw" podUID="bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:13:19.881494 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:19.881056    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb" podUID="373c74f5-bc8f-4938-b64f-f4ab8d6bb993" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:13:25.509962 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:25.509926    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"]
Nov 03 01:13:25.510508 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:25.510236    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc" podUID="5eaa89b0-054c-4c59-a4af-02eefaa98bd6" containerName="istio-proxy" containerID="cri-o://94636314e788d5896234e9d328ac185709a3285f597fe3b013e9b072cc181a29" gracePeriod=30
Nov 03 01:13:29.698769 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:29.698733    5012 generic.go:356] "Generic (PLEG): container finished" podID="373c74f5-bc8f-4938-b64f-f4ab8d6bb993" containerID="71a8b09dd01a0eba38aecd97932469c88015e2b204465eaae74b791e78085709" exitCode=0
Nov 03 01:13:29.699089 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:29.698798    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb" event={"ID":"373c74f5-bc8f-4938-b64f-f4ab8d6bb993","Type":"ContainerDied","Data":"71a8b09dd01a0eba38aecd97932469c88015e2b204465eaae74b791e78085709"}
Nov 03 01:13:29.832190 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:29.832155    5012 patch_prober.go:29] interesting pod/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:13:29.832340 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:29.832226    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc" podUID="5eaa89b0-054c-4c59-a4af-02eefaa98bd6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:13:29.869224 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:29.869202    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb"
Nov 03 01:13:29.897427 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:29.897403    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw"
Nov 03 01:13:29.947837 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:29.947798    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/373c74f5-bc8f-4938-b64f-f4ab8d6bb993-kube-api-access-blns4\" (UniqueName: \"kubernetes.io/projected/373c74f5-bc8f-4938-b64f-f4ab8d6bb993-kube-api-access-blns4\") pod \"373c74f5-bc8f-4938-b64f-f4ab8d6bb993\" (UID: \"373c74f5-bc8f-4938-b64f-f4ab8d6bb993\") "
Nov 03 01:13:29.948034 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:29.947988    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4-kube-api-access-lstlx\" (UniqueName: \"kubernetes.io/projected/bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4-kube-api-access-lstlx\") pod \"bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4\" (UID: \"bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4\") "
Nov 03 01:13:29.948114 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:29.948065    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4-plugins-config-volume\") pod \"bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4\" (UID: \"bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4\") "
Nov 03 01:13:29.948185 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:29.948158    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/373c74f5-bc8f-4938-b64f-f4ab8d6bb993-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373c74f5-bc8f-4938-b64f-f4ab8d6bb993-plugins-config-volume\") pod \"373c74f5-bc8f-4938-b64f-f4ab8d6bb993\" (UID: \"373c74f5-bc8f-4938-b64f-f4ab8d6bb993\") "
Nov 03 01:13:29.948873 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:29.948850    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373c74f5-bc8f-4938-b64f-f4ab8d6bb993-plugins-config-volume" pod "373c74f5-bc8f-4938-b64f-f4ab8d6bb993" (UID: "373c74f5-bc8f-4938-b64f-f4ab8d6bb993"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:13:29.949363 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:29.949337    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4-plugins-config-volume" pod "bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4" (UID: "bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:13:29.951200 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:29.951178    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373c74f5-bc8f-4938-b64f-f4ab8d6bb993-kube-api-access-blns4" pod "373c74f5-bc8f-4938-b64f-f4ab8d6bb993" (UID: "373c74f5-bc8f-4938-b64f-f4ab8d6bb993"). InnerVolumeSpecName "kube-api-access-blns4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:13:29.951425 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:29.951406    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4-kube-api-access-lstlx" pod "bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4" (UID: "bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4"). InnerVolumeSpecName "kube-api-access-lstlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:13:30.049213 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.049188    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lstlx\" (UniqueName: \"kubernetes.io/projected/bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4-kube-api-access-lstlx\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:13:30.049213 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.049212    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:13:30.049421 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.049225    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373c74f5-bc8f-4938-b64f-f4ab8d6bb993-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:13:30.049421 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.049237    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blns4\" (UniqueName: \"kubernetes.io/projected/373c74f5-bc8f-4938-b64f-f4ab8d6bb993-kube-api-access-blns4\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:13:30.709372 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.709337    5012 generic.go:356] "Generic (PLEG): container finished" podID="bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4" containerID="627544b5c7d32d09faf4b80b0fd860d862727c1461723655ee351598b2e01f01" exitCode=0
Nov 03 01:13:30.709820 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.709393    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw"
Nov 03 01:13:30.709820 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.709421    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw" event={"ID":"bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4","Type":"ContainerDied","Data":"627544b5c7d32d09faf4b80b0fd860d862727c1461723655ee351598b2e01f01"}
Nov 03 01:13:30.709820 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.709488    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw" event={"ID":"bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4","Type":"ContainerDied","Data":"7c065384b4612d9b866e7632b092a7646a4dd2c53040ecbbfcf82d302106f4f4"}
Nov 03 01:13:30.709820 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.709516    5012 scope.go:122] "RemoveContainer" containerID="627544b5c7d32d09faf4b80b0fd860d862727c1461723655ee351598b2e01f01"
Nov 03 01:13:30.711865 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.711828    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb" event={"ID":"373c74f5-bc8f-4938-b64f-f4ab8d6bb993","Type":"ContainerDied","Data":"070fcf4f14f738f8802ff25f6ae0e5f63ecd88709168b99dd7be351b53f8c325"}
Nov 03 01:13:30.711865 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.711860    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb"
Nov 03 01:13:30.717276 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.717077    5012 generic.go:356] "Generic (PLEG): container finished" podID="5eaa89b0-054c-4c59-a4af-02eefaa98bd6" containerID="94636314e788d5896234e9d328ac185709a3285f597fe3b013e9b072cc181a29" exitCode=0
Nov 03 01:13:30.717276 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.717150    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc" event={"ID":"5eaa89b0-054c-4c59-a4af-02eefaa98bd6","Type":"ContainerDied","Data":"94636314e788d5896234e9d328ac185709a3285f597fe3b013e9b072cc181a29"}
Nov 03 01:13:30.723727 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.723476    5012 scope.go:122] "RemoveContainer" containerID="627544b5c7d32d09faf4b80b0fd860d862727c1461723655ee351598b2e01f01"
Nov 03 01:13:30.723934 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:13:30.723756    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"627544b5c7d32d09faf4b80b0fd860d862727c1461723655ee351598b2e01f01\": container with ID starting with 627544b5c7d32d09faf4b80b0fd860d862727c1461723655ee351598b2e01f01 not found: ID does not exist" containerID="627544b5c7d32d09faf4b80b0fd860d862727c1461723655ee351598b2e01f01"
Nov 03 01:13:30.723934 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.723789    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"627544b5c7d32d09faf4b80b0fd860d862727c1461723655ee351598b2e01f01"} err="failed to get container status \"627544b5c7d32d09faf4b80b0fd860d862727c1461723655ee351598b2e01f01\": rpc error: code = NotFound desc = could not find container \"627544b5c7d32d09faf4b80b0fd860d862727c1461723655ee351598b2e01f01\": container with ID starting with 627544b5c7d32d09faf4b80b0fd860d862727c1461723655ee351598b2e01f01 not found: ID does not exist"
Nov 03 01:13:30.723934 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.723813    5012 scope.go:122] "RemoveContainer" containerID="71a8b09dd01a0eba38aecd97932469c88015e2b204465eaae74b791e78085709"
Nov 03 01:13:30.735993 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.735966    5012 scope.go:122] "RemoveContainer" containerID="374383517c1fa174e486e662af1576622e10d8b832772519d0b6a999c47de62a"
Nov 03 01:13:30.759561 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.759533    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw"]
Nov 03 01:13:30.769045 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.769017    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw"]
Nov 03 01:13:30.786196 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.786148    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 01:13:30.809335 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.809312    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb"]
Nov 03 01:13:30.814822 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.814802    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-fc9316-6d8f9ff5f9-xfdqb"]
Nov 03 01:13:30.859294 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.859269    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-data\") pod \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") "
Nov 03 01:13:30.859390 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.859312    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istiod-ca-cert\") pod \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") "
Nov 03 01:13:30.859390 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.859348    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-envoy\") pod \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") "
Nov 03 01:13:30.859523 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.859428    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-token\" (UniqueName: \"kubernetes.io/projected/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-token\") pod \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") "
Nov 03 01:13:30.859523 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.859482    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-kube-api-access-6lx5l\" (UniqueName: \"kubernetes.io/projected/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-kube-api-access-6lx5l\") pod \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") "
Nov 03 01:13:30.859523 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.859516    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-workload-certs\") pod \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") "
Nov 03 01:13:30.859694 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.859558    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-data" pod "5eaa89b0-054c-4c59-a4af-02eefaa98bd6" (UID: "5eaa89b0-054c-4c59-a4af-02eefaa98bd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:30.859694 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.859603    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-credential-socket\") pod \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") "
Nov 03 01:13:30.859694 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.859670    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-podinfo\") pod \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") "
Nov 03 01:13:30.859694 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.859693    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-workload-socket\") pod \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\" (UID: \"5eaa89b0-054c-4c59-a4af-02eefaa98bd6\") "
Nov 03 01:13:30.859946 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.859879    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-workload-certs" pod "5eaa89b0-054c-4c59-a4af-02eefaa98bd6" (UID: "5eaa89b0-054c-4c59-a4af-02eefaa98bd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:30.860122 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.860100    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-workload-socket" pod "5eaa89b0-054c-4c59-a4af-02eefaa98bd6" (UID: "5eaa89b0-054c-4c59-a4af-02eefaa98bd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:30.860216 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.860114    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-credential-socket" pod "5eaa89b0-054c-4c59-a4af-02eefaa98bd6" (UID: "5eaa89b0-054c-4c59-a4af-02eefaa98bd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:30.860216 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.860138    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istiod-ca-cert" pod "5eaa89b0-054c-4c59-a4af-02eefaa98bd6" (UID: "5eaa89b0-054c-4c59-a4af-02eefaa98bd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:13:30.860320 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.860279    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:13:30.860320 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.860299    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:13:30.860320 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.860316    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:13:30.860456 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.860330    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:13:30.860456 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.860345    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:13:30.863024 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.862224    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-envoy" pod "5eaa89b0-054c-4c59-a4af-02eefaa98bd6" (UID: "5eaa89b0-054c-4c59-a4af-02eefaa98bd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:30.863024 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.862367    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-token" pod "5eaa89b0-054c-4c59-a4af-02eefaa98bd6" (UID: "5eaa89b0-054c-4c59-a4af-02eefaa98bd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:13:30.863024 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.862785    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-kube-api-access-6lx5l" pod "5eaa89b0-054c-4c59-a4af-02eefaa98bd6" (UID: "5eaa89b0-054c-4c59-a4af-02eefaa98bd6"). InnerVolumeSpecName "kube-api-access-6lx5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:13:30.863752 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.863729    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-podinfo" pod "5eaa89b0-054c-4c59-a4af-02eefaa98bd6" (UID: "5eaa89b0-054c-4c59-a4af-02eefaa98bd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:13:30.880391 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.880358    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-fc9316-7866c6c76c-nxlrw" podUID="bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.80:50052\" within 1s: context deadline exceeded"
Nov 03 01:13:30.880524 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:13:30.880510    5012 logging.go:55] [core] [Channel #15152 SubChannel #15153]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.80:50052", ServerName: "10.132.0.80:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.80:50052: operation was canceled"
Nov 03 01:13:30.961935 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.961883    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:13:30.962058 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.961939    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lx5l\" (UniqueName: \"kubernetes.io/projected/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-kube-api-access-6lx5l\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:13:30.962058 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.961950    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:13:30.962058 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:30.961960    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eaa89b0-054c-4c59-a4af-02eefaa98bd6-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:13:31.243624 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:31.243591    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="373c74f5-bc8f-4938-b64f-f4ab8d6bb993" path="/var/lib/kubelet/pods/373c74f5-bc8f-4938-b64f-f4ab8d6bb993/volumes"
Nov 03 01:13:31.244018 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:31.244003    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4" path="/var/lib/kubelet/pods/bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4/volumes"
Nov 03 01:13:31.727715 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:31.727685    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"
Nov 03 01:13:31.728210 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:31.727790    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc" event={"ID":"5eaa89b0-054c-4c59-a4af-02eefaa98bd6","Type":"ContainerDied","Data":"9680b9ac0de74fdccb474d11ea6642a3db2aa91bbdadac9b57f976c7fa5e3f1d"}
Nov 03 01:13:31.728210 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:31.727850    5012 scope.go:122] "RemoveContainer" containerID="94636314e788d5896234e9d328ac185709a3285f597fe3b013e9b072cc181a29"
Nov 03 01:13:31.795445 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:31.789284    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"]
Nov 03 01:13:31.801746 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:31.801716    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d9e067-openshift-default-7487dcb7c7-lf7gc"]
Nov 03 01:13:33.245522 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:33.245477    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eaa89b0-054c-4c59-a4af-02eefaa98bd6" path="/var/lib/kubelet/pods/5eaa89b0-054c-4c59-a4af-02eefaa98bd6/volumes"
Nov 03 01:13:37.365850 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:13:37.365803    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:13:49.182586 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:49.182551    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7"]
Nov 03 01:13:49.183124 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:49.182754    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7" podUID="2c06dcd4-76c3-4d89-85cd-d14ee32d6567" containerName="maas-api" containerID="cri-o://7de4e5c5c51631b34c367eafb9829d00ae79b3979f19e20688f314559ae9eabc" gracePeriod=30
Nov 03 01:13:49.520171 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:49.520140    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7"
Nov 03 01:13:49.637841 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:49.637798    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-kube-api-access-gqdnw\" (UniqueName: \"kubernetes.io/projected/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-kube-api-access-gqdnw\") pod \"2c06dcd4-76c3-4d89-85cd-d14ee32d6567\" (UID: \"2c06dcd4-76c3-4d89-85cd-d14ee32d6567\") "
Nov 03 01:13:49.638252 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:49.638229    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-openshift-service-ca\") pod \"2c06dcd4-76c3-4d89-85cd-d14ee32d6567\" (UID: \"2c06dcd4-76c3-4d89-85cd-d14ee32d6567\") "
Nov 03 01:13:49.638514 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:49.638491    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-metrics-certs\") pod \"2c06dcd4-76c3-4d89-85cd-d14ee32d6567\" (UID: \"2c06dcd4-76c3-4d89-85cd-d14ee32d6567\") "
Nov 03 01:13:49.638652 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:49.638638    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-maas-api-tls\") pod \"2c06dcd4-76c3-4d89-85cd-d14ee32d6567\" (UID: \"2c06dcd4-76c3-4d89-85cd-d14ee32d6567\") "
Nov 03 01:13:49.655246 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:49.655210    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-openshift-service-ca" pod "2c06dcd4-76c3-4d89-85cd-d14ee32d6567" (UID: "2c06dcd4-76c3-4d89-85cd-d14ee32d6567"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:13:49.692930 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:49.691075    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-maas-api-tls" pod "2c06dcd4-76c3-4d89-85cd-d14ee32d6567" (UID: "2c06dcd4-76c3-4d89-85cd-d14ee32d6567"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:13:49.698927 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:49.694079    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-kube-api-access-gqdnw" pod "2c06dcd4-76c3-4d89-85cd-d14ee32d6567" (UID: "2c06dcd4-76c3-4d89-85cd-d14ee32d6567"). InnerVolumeSpecName "kube-api-access-gqdnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:13:49.698927 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:49.694186    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-metrics-certs" pod "2c06dcd4-76c3-4d89-85cd-d14ee32d6567" (UID: "2c06dcd4-76c3-4d89-85cd-d14ee32d6567"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:13:49.740290 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:49.740227    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:13:49.740290 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:49.740253    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:13:49.740290 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:49.740263    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqdnw\" (UniqueName: \"kubernetes.io/projected/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-kube-api-access-gqdnw\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:13:49.740290 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:49.740274    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c06dcd4-76c3-4d89-85cd-d14ee32d6567-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:13:49.878147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:49.878110    5012 generic.go:356] "Generic (PLEG): container finished" podID="2c06dcd4-76c3-4d89-85cd-d14ee32d6567" containerID="7de4e5c5c51631b34c367eafb9829d00ae79b3979f19e20688f314559ae9eabc" exitCode=0
Nov 03 01:13:49.878330 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:49.878150    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7"
Nov 03 01:13:49.878330 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:49.878189    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7" event={"ID":"2c06dcd4-76c3-4d89-85cd-d14ee32d6567","Type":"ContainerDied","Data":"7de4e5c5c51631b34c367eafb9829d00ae79b3979f19e20688f314559ae9eabc"}
Nov 03 01:13:49.878330 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:49.878244    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7" event={"ID":"2c06dcd4-76c3-4d89-85cd-d14ee32d6567","Type":"ContainerDied","Data":"3ddfbf0c7805452b628d87aa475445f4d8aeb524a90d09250b03c59e0bf20c33"}
Nov 03 01:13:49.878330 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:49.878269    5012 scope.go:122] "RemoveContainer" containerID="7de4e5c5c51631b34c367eafb9829d00ae79b3979f19e20688f314559ae9eabc"
Nov 03 01:13:49.890091 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:49.890074    5012 scope.go:122] "RemoveContainer" containerID="7de4e5c5c51631b34c367eafb9829d00ae79b3979f19e20688f314559ae9eabc"
Nov 03 01:13:49.890344 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:13:49.890328    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7de4e5c5c51631b34c367eafb9829d00ae79b3979f19e20688f314559ae9eabc\": container with ID starting with 7de4e5c5c51631b34c367eafb9829d00ae79b3979f19e20688f314559ae9eabc not found: ID does not exist" containerID="7de4e5c5c51631b34c367eafb9829d00ae79b3979f19e20688f314559ae9eabc"
Nov 03 01:13:49.890456 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:49.890352    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de4e5c5c51631b34c367eafb9829d00ae79b3979f19e20688f314559ae9eabc"} err="failed to get container status \"7de4e5c5c51631b34c367eafb9829d00ae79b3979f19e20688f314559ae9eabc\": rpc error: code = NotFound desc = could not find container \"7de4e5c5c51631b34c367eafb9829d00ae79b3979f19e20688f314559ae9eabc\": container with ID starting with 7de4e5c5c51631b34c367eafb9829d00ae79b3979f19e20688f314559ae9eabc not found: ID does not exist"
Nov 03 01:13:49.927088 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:49.927059    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7"]
Nov 03 01:13:49.937218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:49.937197    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdacdb-7c7f875c89-ww4s7"]
Nov 03 01:13:51.244791 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:13:51.244758    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c06dcd4-76c3-4d89-85cd-d14ee32d6567" path="/var/lib/kubelet/pods/2c06dcd4-76c3-4d89-85cd-d14ee32d6567/volumes"
Nov 03 01:14:07.364684 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:14:07.364644    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:14:15.342631 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:15.342593    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz"]
Nov 03 01:14:15.343306 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:15.342781    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz" podUID="93c536cd-5619-4f22-adb9-b7d1044815f0" containerName="maas-api" containerID="cri-o://9e094b0e024a4ad1849832cd7ac0700b0db4d73659879e35d92eaf927347c494" gracePeriod=30
Nov 03 01:14:15.674583 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:15.674555    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz"
Nov 03 01:14:15.770932 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:15.769203    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c536cd-5619-4f22-adb9-b7d1044815f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c536cd-5619-4f22-adb9-b7d1044815f0-metrics-certs\") pod \"93c536cd-5619-4f22-adb9-b7d1044815f0\" (UID: \"93c536cd-5619-4f22-adb9-b7d1044815f0\") "
Nov 03 01:14:15.770932 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:15.769818    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c536cd-5619-4f22-adb9-b7d1044815f0-kube-api-access-fp5n8\" (UniqueName: \"kubernetes.io/projected/93c536cd-5619-4f22-adb9-b7d1044815f0-kube-api-access-fp5n8\") pod \"93c536cd-5619-4f22-adb9-b7d1044815f0\" (UID: \"93c536cd-5619-4f22-adb9-b7d1044815f0\") "
Nov 03 01:14:15.770932 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:15.770401    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c536cd-5619-4f22-adb9-b7d1044815f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c536cd-5619-4f22-adb9-b7d1044815f0-openshift-service-ca\") pod \"93c536cd-5619-4f22-adb9-b7d1044815f0\" (UID: \"93c536cd-5619-4f22-adb9-b7d1044815f0\") "
Nov 03 01:14:15.770932 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:15.770789    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c536cd-5619-4f22-adb9-b7d1044815f0-openshift-service-ca" pod "93c536cd-5619-4f22-adb9-b7d1044815f0" (UID: "93c536cd-5619-4f22-adb9-b7d1044815f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:14:15.770932 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:15.770880    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c536cd-5619-4f22-adb9-b7d1044815f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c536cd-5619-4f22-adb9-b7d1044815f0-maas-api-tls\") pod \"93c536cd-5619-4f22-adb9-b7d1044815f0\" (UID: \"93c536cd-5619-4f22-adb9-b7d1044815f0\") "
Nov 03 01:14:15.798937 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:15.771903    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c536cd-5619-4f22-adb9-b7d1044815f0-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:14:15.818990 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:15.810921    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c536cd-5619-4f22-adb9-b7d1044815f0-metrics-certs" pod "93c536cd-5619-4f22-adb9-b7d1044815f0" (UID: "93c536cd-5619-4f22-adb9-b7d1044815f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:14:15.837464 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:15.820128    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c536cd-5619-4f22-adb9-b7d1044815f0-maas-api-tls" pod "93c536cd-5619-4f22-adb9-b7d1044815f0" (UID: "93c536cd-5619-4f22-adb9-b7d1044815f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:14:15.837464 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:15.822289    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c536cd-5619-4f22-adb9-b7d1044815f0-kube-api-access-fp5n8" pod "93c536cd-5619-4f22-adb9-b7d1044815f0" (UID: "93c536cd-5619-4f22-adb9-b7d1044815f0"). InnerVolumeSpecName "kube-api-access-fp5n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:14:15.872767 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:15.872733    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c536cd-5619-4f22-adb9-b7d1044815f0-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:14:15.872767 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:15.872765    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp5n8\" (UniqueName: \"kubernetes.io/projected/93c536cd-5619-4f22-adb9-b7d1044815f0-kube-api-access-fp5n8\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:14:15.872767 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:15.872775    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c536cd-5619-4f22-adb9-b7d1044815f0-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:14:16.145991 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:16.144016    5012 generic.go:356] "Generic (PLEG): container finished" podID="93c536cd-5619-4f22-adb9-b7d1044815f0" containerID="9e094b0e024a4ad1849832cd7ac0700b0db4d73659879e35d92eaf927347c494" exitCode=0
Nov 03 01:14:16.145991 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:16.144083    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz"
Nov 03 01:14:16.145991 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:16.144096    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz" event={"ID":"93c536cd-5619-4f22-adb9-b7d1044815f0","Type":"ContainerDied","Data":"9e094b0e024a4ad1849832cd7ac0700b0db4d73659879e35d92eaf927347c494"}
Nov 03 01:14:16.145991 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:16.144152    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz" event={"ID":"93c536cd-5619-4f22-adb9-b7d1044815f0","Type":"ContainerDied","Data":"4abfc1c7ca7217918538a627606e11e5f14b36e3782c32d3933b9525907ad596"}
Nov 03 01:14:16.145991 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:16.144185    5012 scope.go:122] "RemoveContainer" containerID="9e094b0e024a4ad1849832cd7ac0700b0db4d73659879e35d92eaf927347c494"
Nov 03 01:14:16.158831 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:16.158803    5012 scope.go:122] "RemoveContainer" containerID="9e094b0e024a4ad1849832cd7ac0700b0db4d73659879e35d92eaf927347c494"
Nov 03 01:14:16.159278 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:14:16.159254    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e094b0e024a4ad1849832cd7ac0700b0db4d73659879e35d92eaf927347c494\": container with ID starting with 9e094b0e024a4ad1849832cd7ac0700b0db4d73659879e35d92eaf927347c494 not found: ID does not exist" containerID="9e094b0e024a4ad1849832cd7ac0700b0db4d73659879e35d92eaf927347c494"
Nov 03 01:14:16.159365 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:16.159288    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e094b0e024a4ad1849832cd7ac0700b0db4d73659879e35d92eaf927347c494"} err="failed to get container status \"9e094b0e024a4ad1849832cd7ac0700b0db4d73659879e35d92eaf927347c494\": rpc error: code = NotFound desc = could not find container \"9e094b0e024a4ad1849832cd7ac0700b0db4d73659879e35d92eaf927347c494\": container with ID starting with 9e094b0e024a4ad1849832cd7ac0700b0db4d73659879e35d92eaf927347c494 not found: ID does not exist"
Nov 03 01:14:16.192550 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:16.192517    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz"]
Nov 03 01:14:16.202662 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:16.202637    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-065de0-5d4568f94-7lbbz"]
Nov 03 01:14:17.244133 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:17.244100    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93c536cd-5619-4f22-adb9-b7d1044815f0" path="/var/lib/kubelet/pods/93c536cd-5619-4f22-adb9-b7d1044815f0/volumes"
Nov 03 01:14:30.699077 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:30.699039    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn"]
Nov 03 01:14:35.699758 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:35.699712    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn" podUID="36585f09-b213-48d7-b992-e0f2811a7072" containerName="payload-processing" containerID="cri-o://5478b7ab1b64c3a861b34b07ac136b5e31ea526f37be8b4f025cd51a17883d21" gracePeriod=25
Nov 03 01:14:37.362724 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:14:37.362691    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:14:42.287575 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:42.287535    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn" podUID="36585f09-b213-48d7-b992-e0f2811a7072" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:14:47.835662 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:47.835622    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899"]
Nov 03 01:14:47.836334 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:47.835852    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899" podUID="c0136143-9e52-4e16-aa8a-0e190e0239e2" containerName="maas-api" containerID="cri-o://2aa0d72633a420ad684bbbb659bc2edf074c96e91cb1e7094aab345cc340bf63" gracePeriod=30
Nov 03 01:14:48.154461 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:48.154420    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899"
Nov 03 01:14:48.277670 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:48.277637    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0136143-9e52-4e16-aa8a-0e190e0239e2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0136143-9e52-4e16-aa8a-0e190e0239e2-openshift-service-ca\") pod \"c0136143-9e52-4e16-aa8a-0e190e0239e2\" (UID: \"c0136143-9e52-4e16-aa8a-0e190e0239e2\") "
Nov 03 01:14:48.277875 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:48.277677    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0136143-9e52-4e16-aa8a-0e190e0239e2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0136143-9e52-4e16-aa8a-0e190e0239e2-metrics-certs\") pod \"c0136143-9e52-4e16-aa8a-0e190e0239e2\" (UID: \"c0136143-9e52-4e16-aa8a-0e190e0239e2\") "
Nov 03 01:14:48.277875 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:48.277813    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0136143-9e52-4e16-aa8a-0e190e0239e2-kube-api-access-c7686\" (UniqueName: \"kubernetes.io/projected/c0136143-9e52-4e16-aa8a-0e190e0239e2-kube-api-access-c7686\") pod \"c0136143-9e52-4e16-aa8a-0e190e0239e2\" (UID: \"c0136143-9e52-4e16-aa8a-0e190e0239e2\") "
Nov 03 01:14:48.277875 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:48.277865    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0136143-9e52-4e16-aa8a-0e190e0239e2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0136143-9e52-4e16-aa8a-0e190e0239e2-maas-api-tls\") pod \"c0136143-9e52-4e16-aa8a-0e190e0239e2\" (UID: \"c0136143-9e52-4e16-aa8a-0e190e0239e2\") "
Nov 03 01:14:48.278116 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:48.278085    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0136143-9e52-4e16-aa8a-0e190e0239e2-openshift-service-ca" pod "c0136143-9e52-4e16-aa8a-0e190e0239e2" (UID: "c0136143-9e52-4e16-aa8a-0e190e0239e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:14:48.278457 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:48.278436    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0136143-9e52-4e16-aa8a-0e190e0239e2-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:14:48.280076 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:48.280051    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0136143-9e52-4e16-aa8a-0e190e0239e2-maas-api-tls" pod "c0136143-9e52-4e16-aa8a-0e190e0239e2" (UID: "c0136143-9e52-4e16-aa8a-0e190e0239e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:14:48.280169 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:48.280103    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0136143-9e52-4e16-aa8a-0e190e0239e2-kube-api-access-c7686" pod "c0136143-9e52-4e16-aa8a-0e190e0239e2" (UID: "c0136143-9e52-4e16-aa8a-0e190e0239e2"). InnerVolumeSpecName "kube-api-access-c7686". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:14:48.280169 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:48.280105    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0136143-9e52-4e16-aa8a-0e190e0239e2-metrics-certs" pod "c0136143-9e52-4e16-aa8a-0e190e0239e2" (UID: "c0136143-9e52-4e16-aa8a-0e190e0239e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:14:48.388945 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:48.384982    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7686\" (UniqueName: \"kubernetes.io/projected/c0136143-9e52-4e16-aa8a-0e190e0239e2-kube-api-access-c7686\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:14:48.388945 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:48.385016    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0136143-9e52-4e16-aa8a-0e190e0239e2-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:14:48.388945 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:48.385030    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0136143-9e52-4e16-aa8a-0e190e0239e2-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:14:48.403970 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:48.402584    5012 generic.go:356] "Generic (PLEG): container finished" podID="c0136143-9e52-4e16-aa8a-0e190e0239e2" containerID="2aa0d72633a420ad684bbbb659bc2edf074c96e91cb1e7094aab345cc340bf63" exitCode=0
Nov 03 01:14:48.403970 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:48.402635    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899" event={"ID":"c0136143-9e52-4e16-aa8a-0e190e0239e2","Type":"ContainerDied","Data":"2aa0d72633a420ad684bbbb659bc2edf074c96e91cb1e7094aab345cc340bf63"}
Nov 03 01:14:48.403970 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:48.402666    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899"
Nov 03 01:14:48.403970 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:48.402736    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899" event={"ID":"c0136143-9e52-4e16-aa8a-0e190e0239e2","Type":"ContainerDied","Data":"714569585efc7580e46640157aa1c702082cd70126ecee711aaab6ded24aa046"}
Nov 03 01:14:48.403970 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:48.402772    5012 scope.go:122] "RemoveContainer" containerID="2aa0d72633a420ad684bbbb659bc2edf074c96e91cb1e7094aab345cc340bf63"
Nov 03 01:14:48.418587 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:48.418562    5012 scope.go:122] "RemoveContainer" containerID="2aa0d72633a420ad684bbbb659bc2edf074c96e91cb1e7094aab345cc340bf63"
Nov 03 01:14:48.418901 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:14:48.418878    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aa0d72633a420ad684bbbb659bc2edf074c96e91cb1e7094aab345cc340bf63\": container with ID starting with 2aa0d72633a420ad684bbbb659bc2edf074c96e91cb1e7094aab345cc340bf63 not found: ID does not exist" containerID="2aa0d72633a420ad684bbbb659bc2edf074c96e91cb1e7094aab345cc340bf63"
Nov 03 01:14:48.419022 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:48.418925    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aa0d72633a420ad684bbbb659bc2edf074c96e91cb1e7094aab345cc340bf63"} err="failed to get container status \"2aa0d72633a420ad684bbbb659bc2edf074c96e91cb1e7094aab345cc340bf63\": rpc error: code = NotFound desc = could not find container \"2aa0d72633a420ad684bbbb659bc2edf074c96e91cb1e7094aab345cc340bf63\": container with ID starting with 2aa0d72633a420ad684bbbb659bc2edf074c96e91cb1e7094aab345cc340bf63 not found: ID does not exist"
Nov 03 01:14:48.450473 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:48.450452    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899"]
Nov 03 01:14:48.453970 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:48.453948    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ef1b6-8658c95ffb-58899"]
Nov 03 01:14:49.246214 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:49.246157    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0136143-9e52-4e16-aa8a-0e190e0239e2" path="/var/lib/kubelet/pods/c0136143-9e52-4e16-aa8a-0e190e0239e2/volumes"
Nov 03 01:14:52.286763 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:52.286727    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn" podUID="36585f09-b213-48d7-b992-e0f2811a7072" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:14:54.498144 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:54.498103    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"]
Nov 03 01:14:54.498693 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:54.498374    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd" podUID="dbbcfc46-aa01-443c-a475-a757649f86ce" containerName="istio-proxy" containerID="cri-o://c83cb7eaf2e28ef1ab4cf8455339c11b76f959fa66a8f50b1d84f303f7c07188" gracePeriod=30
Nov 03 01:14:55.987897 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:55.987868    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn"
Nov 03 01:14:56.175348 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:56.175302    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36585f09-b213-48d7-b992-e0f2811a7072-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36585f09-b213-48d7-b992-e0f2811a7072-plugins-config-volume\") pod \"36585f09-b213-48d7-b992-e0f2811a7072\" (UID: \"36585f09-b213-48d7-b992-e0f2811a7072\") "
Nov 03 01:14:56.175550 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:56.175421    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36585f09-b213-48d7-b992-e0f2811a7072-kube-api-access-s9jfj\" (UniqueName: \"kubernetes.io/projected/36585f09-b213-48d7-b992-e0f2811a7072-kube-api-access-s9jfj\") pod \"36585f09-b213-48d7-b992-e0f2811a7072\" (UID: \"36585f09-b213-48d7-b992-e0f2811a7072\") "
Nov 03 01:14:56.175683 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:56.175658    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36585f09-b213-48d7-b992-e0f2811a7072-plugins-config-volume" pod "36585f09-b213-48d7-b992-e0f2811a7072" (UID: "36585f09-b213-48d7-b992-e0f2811a7072"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:14:56.175837 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:56.175821    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36585f09-b213-48d7-b992-e0f2811a7072-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:14:56.177487 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:56.177461    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36585f09-b213-48d7-b992-e0f2811a7072-kube-api-access-s9jfj" pod "36585f09-b213-48d7-b992-e0f2811a7072" (UID: "36585f09-b213-48d7-b992-e0f2811a7072"). InnerVolumeSpecName "kube-api-access-s9jfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:14:56.276852 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:56.276768    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9jfj\" (UniqueName: \"kubernetes.io/projected/36585f09-b213-48d7-b992-e0f2811a7072-kube-api-access-s9jfj\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:14:56.471935 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:56.471486    5012 generic.go:356] "Generic (PLEG): container finished" podID="36585f09-b213-48d7-b992-e0f2811a7072" containerID="5478b7ab1b64c3a861b34b07ac136b5e31ea526f37be8b4f025cd51a17883d21" exitCode=0
Nov 03 01:14:56.471935 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:56.471558    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn"
Nov 03 01:14:56.471935 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:56.471584    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn" event={"ID":"36585f09-b213-48d7-b992-e0f2811a7072","Type":"ContainerDied","Data":"5478b7ab1b64c3a861b34b07ac136b5e31ea526f37be8b4f025cd51a17883d21"}
Nov 03 01:14:56.471935 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:56.471659    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn" event={"ID":"36585f09-b213-48d7-b992-e0f2811a7072","Type":"ContainerDied","Data":"c6f7dfead6387e4ad8fc41f1bf80f1aa0740ee93ef0922c5cff044b256377801"}
Nov 03 01:14:56.471935 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:56.471698    5012 scope.go:122] "RemoveContainer" containerID="5478b7ab1b64c3a861b34b07ac136b5e31ea526f37be8b4f025cd51a17883d21"
Nov 03 01:14:56.482578 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:56.482557    5012 scope.go:122] "RemoveContainer" containerID="5478b7ab1b64c3a861b34b07ac136b5e31ea526f37be8b4f025cd51a17883d21"
Nov 03 01:14:56.482870 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:14:56.482844    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5478b7ab1b64c3a861b34b07ac136b5e31ea526f37be8b4f025cd51a17883d21\": container with ID starting with 5478b7ab1b64c3a861b34b07ac136b5e31ea526f37be8b4f025cd51a17883d21 not found: ID does not exist" containerID="5478b7ab1b64c3a861b34b07ac136b5e31ea526f37be8b4f025cd51a17883d21"
Nov 03 01:14:56.483022 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:56.482876    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5478b7ab1b64c3a861b34b07ac136b5e31ea526f37be8b4f025cd51a17883d21"} err="failed to get container status \"5478b7ab1b64c3a861b34b07ac136b5e31ea526f37be8b4f025cd51a17883d21\": rpc error: code = NotFound desc = could not find container \"5478b7ab1b64c3a861b34b07ac136b5e31ea526f37be8b4f025cd51a17883d21\": container with ID starting with 5478b7ab1b64c3a861b34b07ac136b5e31ea526f37be8b4f025cd51a17883d21 not found: ID does not exist"
Nov 03 01:14:56.519506 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:56.519473    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn"]
Nov 03 01:14:56.525825 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:56.525795    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7ef1b6-5d8d55789c-5hhmn"]
Nov 03 01:14:57.245712 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:57.245667    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36585f09-b213-48d7-b992-e0f2811a7072" path="/var/lib/kubelet/pods/36585f09-b213-48d7-b992-e0f2811a7072/volumes"
Nov 03 01:14:59.788448 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:59.788420    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 01:14:59.922051 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:59.922016    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-workload-socket\") pod \"dbbcfc46-aa01-443c-a475-a757649f86ce\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") "
Nov 03 01:14:59.922255 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:59.922069    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbbcfc46-aa01-443c-a475-a757649f86ce-kube-api-access-f4tm9\" (UniqueName: \"kubernetes.io/projected/dbbcfc46-aa01-443c-a475-a757649f86ce-kube-api-access-f4tm9\") pod \"dbbcfc46-aa01-443c-a475-a757649f86ce\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") "
Nov 03 01:14:59.922255 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:59.922108    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-podinfo\") pod \"dbbcfc46-aa01-443c-a475-a757649f86ce\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") "
Nov 03 01:14:59.922255 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:59.922177    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-credential-socket\") pod \"dbbcfc46-aa01-443c-a475-a757649f86ce\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") "
Nov 03 01:14:59.922255 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:59.922209    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbbcfc46-aa01-443c-a475-a757649f86ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbbcfc46-aa01-443c-a475-a757649f86ce-istiod-ca-cert\") pod \"dbbcfc46-aa01-443c-a475-a757649f86ce\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") "
Nov 03 01:14:59.922255 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:59.922230    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-token\" (UniqueName: \"kubernetes.io/projected/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-token\") pod \"dbbcfc46-aa01-443c-a475-a757649f86ce\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") "
Nov 03 01:14:59.922255 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:59.922250    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-data\") pod \"dbbcfc46-aa01-443c-a475-a757649f86ce\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") "
Nov 03 01:14:59.922565 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:59.922293    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-workload-certs\") pod \"dbbcfc46-aa01-443c-a475-a757649f86ce\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") "
Nov 03 01:14:59.922565 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:59.922337    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-envoy\") pod \"dbbcfc46-aa01-443c-a475-a757649f86ce\" (UID: \"dbbcfc46-aa01-443c-a475-a757649f86ce\") "
Nov 03 01:14:59.922565 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:59.922337    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-workload-socket" pod "dbbcfc46-aa01-443c-a475-a757649f86ce" (UID: "dbbcfc46-aa01-443c-a475-a757649f86ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:59.922733 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:59.922536    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-credential-socket" pod "dbbcfc46-aa01-443c-a475-a757649f86ce" (UID: "dbbcfc46-aa01-443c-a475-a757649f86ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:59.922882 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:59.922783    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-data" pod "dbbcfc46-aa01-443c-a475-a757649f86ce" (UID: "dbbcfc46-aa01-443c-a475-a757649f86ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:59.922882 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:59.922801    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-workload-certs" pod "dbbcfc46-aa01-443c-a475-a757649f86ce" (UID: "dbbcfc46-aa01-443c-a475-a757649f86ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:59.922882 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:59.922855    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:14:59.922882 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:59.922879    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:14:59.923163 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:59.922850    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbbcfc46-aa01-443c-a475-a757649f86ce-istiod-ca-cert" pod "dbbcfc46-aa01-443c-a475-a757649f86ce" (UID: "dbbcfc46-aa01-443c-a475-a757649f86ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:14:59.924587 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:59.924562    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-podinfo" pod "dbbcfc46-aa01-443c-a475-a757649f86ce" (UID: "dbbcfc46-aa01-443c-a475-a757649f86ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:14:59.924685 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:59.924600    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-token" pod "dbbcfc46-aa01-443c-a475-a757649f86ce" (UID: "dbbcfc46-aa01-443c-a475-a757649f86ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:14:59.924685 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:59.924626    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbbcfc46-aa01-443c-a475-a757649f86ce-kube-api-access-f4tm9" pod "dbbcfc46-aa01-443c-a475-a757649f86ce" (UID: "dbbcfc46-aa01-443c-a475-a757649f86ce"). InnerVolumeSpecName "kube-api-access-f4tm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:14:59.924685 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:59.924628    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-envoy" pod "dbbcfc46-aa01-443c-a475-a757649f86ce" (UID: "dbbcfc46-aa01-443c-a475-a757649f86ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:59.978275 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:14:59.978246    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"]
Nov 03 01:15:00.024126 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.024068    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4tm9\" (UniqueName: \"kubernetes.io/projected/dbbcfc46-aa01-443c-a475-a757649f86ce-kube-api-access-f4tm9\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:00.024126 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.024094    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:00.024126 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.024106    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbbcfc46-aa01-443c-a475-a757649f86ce-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:00.024126 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.024116    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:00.024126 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.024126    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:00.024369 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.024139    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:00.024369 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.024150    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbbcfc46-aa01-443c-a475-a757649f86ce-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:00.211804 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.211769    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g"]
Nov 03 01:15:00.212943 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.212899    5012 cpu_manager.go:414] "containerMap: removing container" podUID="5d0c60ba-1ab0-4a70-82b9-78580583e5cf" containerName="payload-processing"
Nov 03 01:15:00.213038 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.212947    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0c60ba-1ab0-4a70-82b9-78580583e5cf" containerName="payload-processing"
Nov 03 01:15:00.213038 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.212966    5012 cpu_manager.go:414] "containerMap: removing container" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main"
Nov 03 01:15:00.213038 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.212975    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main"
Nov 03 01:15:00.213038 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.212993    5012 cpu_manager.go:414] "containerMap: removing container" podUID="bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4" containerName="payload-pre-processing"
Nov 03 01:15:00.213038 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213001    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4" containerName="payload-pre-processing"
Nov 03 01:15:00.213038 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213013    5012 cpu_manager.go:414] "containerMap: removing container" podUID="dbbcfc46-aa01-443c-a475-a757649f86ce" containerName="istio-proxy"
Nov 03 01:15:00.213038 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213021    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbbcfc46-aa01-443c-a475-a757649f86ce" containerName="istio-proxy"
Nov 03 01:15:00.213038 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213037    5012 cpu_manager.go:414] "containerMap: removing container" podUID="ce10e6fa-6b29-4edc-ac96-e81e7c7eebab" containerName="maas-api"
Nov 03 01:15:00.213304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213045    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce10e6fa-6b29-4edc-ac96-e81e7c7eebab" containerName="maas-api"
Nov 03 01:15:00.213304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213065    5012 cpu_manager.go:414] "containerMap: removing container" podUID="93c536cd-5619-4f22-adb9-b7d1044815f0" containerName="maas-api"
Nov 03 01:15:00.213304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213073    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c536cd-5619-4f22-adb9-b7d1044815f0" containerName="maas-api"
Nov 03 01:15:00.213304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213083    5012 cpu_manager.go:414] "containerMap: removing container" podUID="0b949944-2ed2-4a75-82d1-a2e2854e3355" containerName="istio-proxy"
Nov 03 01:15:00.213304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213090    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b949944-2ed2-4a75-82d1-a2e2854e3355" containerName="istio-proxy"
Nov 03 01:15:00.213304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213103    5012 cpu_manager.go:414] "containerMap: removing container" podUID="5eaa89b0-054c-4c59-a4af-02eefaa98bd6" containerName="istio-proxy"
Nov 03 01:15:00.213304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213110    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eaa89b0-054c-4c59-a4af-02eefaa98bd6" containerName="istio-proxy"
Nov 03 01:15:00.213304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213121    5012 cpu_manager.go:414] "containerMap: removing container" podUID="2a72ca88-0371-4fd7-ab82-3019bd21d9f6" containerName="maas-api"
Nov 03 01:15:00.213304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213129    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a72ca88-0371-4fd7-ab82-3019bd21d9f6" containerName="maas-api"
Nov 03 01:15:00.213304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213141    5012 cpu_manager.go:414] "containerMap: removing container" podUID="36585f09-b213-48d7-b992-e0f2811a7072" containerName="payload-processing"
Nov 03 01:15:00.213304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213148    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36585f09-b213-48d7-b992-e0f2811a7072" containerName="payload-processing"
Nov 03 01:15:00.213304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213163    5012 cpu_manager.go:414] "containerMap: removing container" podUID="fcb95377-3137-4676-b9b5-c464755a097f" containerName="payload-pre-processing"
Nov 03 01:15:00.213304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213170    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcb95377-3137-4676-b9b5-c464755a097f" containerName="payload-pre-processing"
Nov 03 01:15:00.213304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213183    5012 cpu_manager.go:414] "containerMap: removing container" podUID="373c74f5-bc8f-4938-b64f-f4ab8d6bb993" containerName="payload-processing"
Nov 03 01:15:00.213304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213190    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="373c74f5-bc8f-4938-b64f-f4ab8d6bb993" containerName="payload-processing"
Nov 03 01:15:00.213304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213200    5012 cpu_manager.go:414] "containerMap: removing container" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main"
Nov 03 01:15:00.213304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213207    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main"
Nov 03 01:15:00.213304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213234    5012 cpu_manager.go:414] "containerMap: removing container" podUID="5795e973-761d-43af-91ea-0451024c0d42" containerName="istio-proxy"
Nov 03 01:15:00.213304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213242    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5795e973-761d-43af-91ea-0451024c0d42" containerName="istio-proxy"
Nov 03 01:15:00.213304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213253    5012 cpu_manager.go:414] "containerMap: removing container" podUID="5795e973-761d-43af-91ea-0451024c0d42" containerName="istio-proxy"
Nov 03 01:15:00.213304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213260    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5795e973-761d-43af-91ea-0451024c0d42" containerName="istio-proxy"
Nov 03 01:15:00.213304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213273    5012 cpu_manager.go:414] "containerMap: removing container" podUID="cafedf4f-719f-4e62-9f27-28f2e2017131" containerName="payload-processing"
Nov 03 01:15:00.213304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213280    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cafedf4f-719f-4e62-9f27-28f2e2017131" containerName="payload-processing"
Nov 03 01:15:00.213304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213296    5012 cpu_manager.go:414] "containerMap: removing container" podUID="5cda386d-6a1d-4fd9-95c5-1a68750e10b0" containerName="maas-api"
Nov 03 01:15:00.213304 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213303    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cda386d-6a1d-4fd9-95c5-1a68750e10b0" containerName="maas-api"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213317    5012 cpu_manager.go:414] "containerMap: removing container" podUID="c0136143-9e52-4e16-aa8a-0e190e0239e2" containerName="maas-api"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213325    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0136143-9e52-4e16-aa8a-0e190e0239e2" containerName="maas-api"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213334    5012 cpu_manager.go:414] "containerMap: removing container" podUID="e9bafb53-588d-4773-8c0b-aa8682c19d37" containerName="maas-api"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213342    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9bafb53-588d-4773-8c0b-aa8682c19d37" containerName="maas-api"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213356    5012 cpu_manager.go:414] "containerMap: removing container" podUID="d4aa636c-5490-4e79-9b11-3284af3e64a6" containerName="payload-pre-processing"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213364    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4aa636c-5490-4e79-9b11-3284af3e64a6" containerName="payload-pre-processing"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213376    5012 cpu_manager.go:414] "containerMap: removing container" podUID="16461b4f-b9e0-4313-afbd-15c34c4bd9dd" containerName="maas-api"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213384    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16461b4f-b9e0-4313-afbd-15c34c4bd9dd" containerName="maas-api"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213392    5012 cpu_manager.go:414] "containerMap: removing container" podUID="4233c829-fa1c-443e-b7b6-9522c977fa98" containerName="istio-proxy"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213399    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4233c829-fa1c-443e-b7b6-9522c977fa98" containerName="istio-proxy"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213412    5012 cpu_manager.go:414] "containerMap: removing container" podUID="2c06dcd4-76c3-4d89-85cd-d14ee32d6567" containerName="maas-api"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213419    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c06dcd4-76c3-4d89-85cd-d14ee32d6567" containerName="maas-api"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213432    5012 cpu_manager.go:414] "containerMap: removing container" podUID="a314ceb2-941e-4c6d-8054-8c3bd7155692" containerName="payload-processing"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213439    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a314ceb2-941e-4c6d-8054-8c3bd7155692" containerName="payload-processing"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213450    5012 cpu_manager.go:414] "containerMap: removing container" podUID="2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64" containerName="maas-api"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213458    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64" containerName="maas-api"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213594    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="36585f09-b213-48d7-b992-e0f2811a7072" containerName="payload-processing"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213608    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcb95377-3137-4676-b9b5-c464755a097f" containerName="payload-pre-processing"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213618    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="373c74f5-bc8f-4938-b64f-f4ab8d6bb993" containerName="payload-processing"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213630    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="170314c3-bfce-424b-bba2-3dc03d1b80e4" containerName="main"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213641    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eaa89b0-054c-4c59-a4af-02eefaa98bd6" containerName="istio-proxy"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213650    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="5795e973-761d-43af-91ea-0451024c0d42" containerName="istio-proxy"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213659    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="5795e973-761d-43af-91ea-0451024c0d42" containerName="istio-proxy"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213667    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="cafedf4f-719f-4e62-9f27-28f2e2017131" containerName="payload-processing"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213678    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cda386d-6a1d-4fd9-95c5-1a68750e10b0" containerName="maas-api"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213688    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0136143-9e52-4e16-aa8a-0e190e0239e2" containerName="maas-api"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213697    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9bafb53-588d-4773-8c0b-aa8682c19d37" containerName="maas-api"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213707    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4aa636c-5490-4e79-9b11-3284af3e64a6" containerName="payload-pre-processing"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213717    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="209af59a-bfd0-4aab-b2bf-7c789190f57e" containerName="istio-proxy"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213725    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="16461b4f-b9e0-4313-afbd-15c34c4bd9dd" containerName="maas-api"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213734    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbbcfc46-aa01-443c-a475-a757649f86ce" containerName="istio-proxy"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213742    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c06dcd4-76c3-4d89-85cd-d14ee32d6567" containerName="maas-api"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213751    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="a314ceb2-941e-4c6d-8054-8c3bd7155692" containerName="payload-processing"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213761    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f4e5fb2-d1a9-4070-8e9b-3c60656bfa64" containerName="maas-api"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213769    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0c60ba-1ab0-4a70-82b9-78580583e5cf" containerName="payload-processing"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213780    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9b22dc8-f6f1-4c9f-b3ec-28b6ada72095" containerName="main"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213790    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc6d7c40-fd39-4fcb-920c-cabd4d0b08d4" containerName="payload-pre-processing"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213798    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbbcfc46-aa01-443c-a475-a757649f86ce" containerName="istio-proxy"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213808    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce10e6fa-6b29-4edc-ac96-e81e7c7eebab" containerName="maas-api"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213818    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c536cd-5619-4f22-adb9-b7d1044815f0" containerName="maas-api"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213827    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eaa89b0-054c-4c59-a4af-02eefaa98bd6" containerName="istio-proxy"
Nov 03 01:15:00.214455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.213836    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a72ca88-0371-4fd7-ab82-3019bd21d9f6" containerName="maas-api"
Nov 03 01:15:00.218159 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.218132    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv"]
Nov 03 01:15:00.218330 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.218274    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g"
Nov 03 01:15:00.218817 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.218796    5012 cpu_manager.go:414] "containerMap: removing container" podUID="dbbcfc46-aa01-443c-a475-a757649f86ce" containerName="istio-proxy"
Nov 03 01:15:00.218817 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.218816    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbbcfc46-aa01-443c-a475-a757649f86ce" containerName="istio-proxy"
Nov 03 01:15:00.219020 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.218835    5012 cpu_manager.go:414] "containerMap: removing container" podUID="5eaa89b0-054c-4c59-a4af-02eefaa98bd6" containerName="istio-proxy"
Nov 03 01:15:00.219020 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.218840    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eaa89b0-054c-4c59-a4af-02eefaa98bd6" containerName="istio-proxy"
Nov 03 01:15:00.222166 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.222145    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv"
Nov 03 01:15:00.238481 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.238410    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv"]
Nov 03 01:15:00.243440 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.243417    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g"]
Nov 03 01:15:00.287036 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.286967    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:15:00.327462 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.327400    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c993ca-dfa2-422f-bb9b-21af3056b405-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv\" (UID: \"c3c993ca-dfa2-422f-bb9b-21af3056b405\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv"
Nov 03 01:15:00.327625 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.327530    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plnb8\" (UniqueName: \"kubernetes.io/projected/8c26a270-d736-4678-9ff8-bfabcb06fe44-kube-api-access-plnb8\") pod \"maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g\" (UID: \"8c26a270-d736-4678-9ff8-bfabcb06fe44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g"
Nov 03 01:15:00.327693 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.327640    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c26a270-d736-4678-9ff8-bfabcb06fe44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g\" (UID: \"8c26a270-d736-4678-9ff8-bfabcb06fe44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g"
Nov 03 01:15:00.327756 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.327705    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfcnw\" (UniqueName: \"kubernetes.io/projected/c3c993ca-dfa2-422f-bb9b-21af3056b405-kube-api-access-mfcnw\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv\" (UID: \"c3c993ca-dfa2-422f-bb9b-21af3056b405\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv"
Nov 03 01:15:00.428611 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.428581    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c26a270-d736-4678-9ff8-bfabcb06fe44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g\" (UID: \"8c26a270-d736-4678-9ff8-bfabcb06fe44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g"
Nov 03 01:15:00.428760 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.428635    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfcnw\" (UniqueName: \"kubernetes.io/projected/c3c993ca-dfa2-422f-bb9b-21af3056b405-kube-api-access-mfcnw\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv\" (UID: \"c3c993ca-dfa2-422f-bb9b-21af3056b405\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv"
Nov 03 01:15:00.428760 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.428724    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c993ca-dfa2-422f-bb9b-21af3056b405-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv\" (UID: \"c3c993ca-dfa2-422f-bb9b-21af3056b405\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv"
Nov 03 01:15:00.428937 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.428895    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plnb8\" (UniqueName: \"kubernetes.io/projected/8c26a270-d736-4678-9ff8-bfabcb06fe44-kube-api-access-plnb8\") pod \"maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g\" (UID: \"8c26a270-d736-4678-9ff8-bfabcb06fe44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g"
Nov 03 01:15:00.429401 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.429376    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c26a270-d736-4678-9ff8-bfabcb06fe44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g\" (UID: \"8c26a270-d736-4678-9ff8-bfabcb06fe44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g"
Nov 03 01:15:00.429488 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.429472    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c993ca-dfa2-422f-bb9b-21af3056b405-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv\" (UID: \"c3c993ca-dfa2-422f-bb9b-21af3056b405\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv"
Nov 03 01:15:00.446420 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.446393    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfcnw\" (UniqueName: \"kubernetes.io/projected/c3c993ca-dfa2-422f-bb9b-21af3056b405-kube-api-access-mfcnw\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv\" (UID: \"c3c993ca-dfa2-422f-bb9b-21af3056b405\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv"
Nov 03 01:15:00.446511 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.446459    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plnb8\" (UniqueName: \"kubernetes.io/projected/8c26a270-d736-4678-9ff8-bfabcb06fe44-kube-api-access-plnb8\") pod \"maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g\" (UID: \"8c26a270-d736-4678-9ff8-bfabcb06fe44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g"
Nov 03 01:15:00.515432 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.515403    5012 generic.go:356] "Generic (PLEG): container finished" podID="dbbcfc46-aa01-443c-a475-a757649f86ce" containerID="c83cb7eaf2e28ef1ab4cf8455339c11b76f959fa66a8f50b1d84f303f7c07188" exitCode=0
Nov 03 01:15:00.515554 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.515470    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"
Nov 03 01:15:00.515554 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.515494    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd" event={"ID":"dbbcfc46-aa01-443c-a475-a757649f86ce","Type":"ContainerDied","Data":"c83cb7eaf2e28ef1ab4cf8455339c11b76f959fa66a8f50b1d84f303f7c07188"}
Nov 03 01:15:00.515684 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.515586    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd" event={"ID":"dbbcfc46-aa01-443c-a475-a757649f86ce","Type":"ContainerDied","Data":"a6867856ef617c362916988988eec8614b746fe8e29a4040e40badc2c30af7be"}
Nov 03 01:15:00.515684 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.515623    5012 scope.go:122] "RemoveContainer" containerID="c83cb7eaf2e28ef1ab4cf8455339c11b76f959fa66a8f50b1d84f303f7c07188"
Nov 03 01:15:00.526129 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.526106    5012 scope.go:122] "RemoveContainer" containerID="bc2f9bac5940460d145b142b868b1dcd75e0d4c5bdd5f1542e0d841235297be2"
Nov 03 01:15:00.540089 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.540041    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g"
Nov 03 01:15:00.540774 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.540757    5012 scope.go:122] "RemoveContainer" containerID="c83cb7eaf2e28ef1ab4cf8455339c11b76f959fa66a8f50b1d84f303f7c07188"
Nov 03 01:15:00.541020 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:15:00.540999    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c83cb7eaf2e28ef1ab4cf8455339c11b76f959fa66a8f50b1d84f303f7c07188\": container with ID starting with c83cb7eaf2e28ef1ab4cf8455339c11b76f959fa66a8f50b1d84f303f7c07188 not found: ID does not exist" containerID="c83cb7eaf2e28ef1ab4cf8455339c11b76f959fa66a8f50b1d84f303f7c07188"
Nov 03 01:15:00.541076 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.541027    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c83cb7eaf2e28ef1ab4cf8455339c11b76f959fa66a8f50b1d84f303f7c07188"} err="failed to get container status \"c83cb7eaf2e28ef1ab4cf8455339c11b76f959fa66a8f50b1d84f303f7c07188\": rpc error: code = NotFound desc = could not find container \"c83cb7eaf2e28ef1ab4cf8455339c11b76f959fa66a8f50b1d84f303f7c07188\": container with ID starting with c83cb7eaf2e28ef1ab4cf8455339c11b76f959fa66a8f50b1d84f303f7c07188 not found: ID does not exist"
Nov 03 01:15:00.541076 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.541043    5012 scope.go:122] "RemoveContainer" containerID="bc2f9bac5940460d145b142b868b1dcd75e0d4c5bdd5f1542e0d841235297be2"
Nov 03 01:15:00.541330 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:15:00.541311    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc2f9bac5940460d145b142b868b1dcd75e0d4c5bdd5f1542e0d841235297be2\": container with ID starting with bc2f9bac5940460d145b142b868b1dcd75e0d4c5bdd5f1542e0d841235297be2 not found: ID does not exist" containerID="bc2f9bac5940460d145b142b868b1dcd75e0d4c5bdd5f1542e0d841235297be2"
Nov 03 01:15:00.541380 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.541335    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc2f9bac5940460d145b142b868b1dcd75e0d4c5bdd5f1542e0d841235297be2"} err="failed to get container status \"bc2f9bac5940460d145b142b868b1dcd75e0d4c5bdd5f1542e0d841235297be2\": rpc error: code = NotFound desc = could not find container \"bc2f9bac5940460d145b142b868b1dcd75e0d4c5bdd5f1542e0d841235297be2\": container with ID starting with bc2f9bac5940460d145b142b868b1dcd75e0d4c5bdd5f1542e0d841235297be2 not found: ID does not exist"
Nov 03 01:15:00.560627 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.560605    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv"
Nov 03 01:15:00.586693 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.586662    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"]
Nov 03 01:15:00.611934 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.609734    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7ef1b6-openshift-default-9f5d75d5f-6m9vd"]
Nov 03 01:15:00.843439 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.843413    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g"]
Nov 03 01:15:00.844259 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:15:00.844237    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c26a270_d736_4678_9ff8_bfabcb06fe44.slice/crio-fb419c0b12d3e078cbcf22db78f382e45d4f887855e1163e134cfcb448ca125d WatchSource:0}: Error finding container fb419c0b12d3e078cbcf22db78f382e45d4f887855e1163e134cfcb448ca125d: Status 404 returned error can't find the container with id fb419c0b12d3e078cbcf22db78f382e45d4f887855e1163e134cfcb448ca125d
Nov 03 01:15:00.868073 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:00.868050    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv"]
Nov 03 01:15:00.868263 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:15:00.868241    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c993ca_dfa2_422f_bb9b_21af3056b405.slice/crio-fb51dc632eea870fa6b110d9994f576ffa506fe450c0ee6619a4cc188adec581 WatchSource:0}: Error finding container fb51dc632eea870fa6b110d9994f576ffa506fe450c0ee6619a4cc188adec581: Status 404 returned error can't find the container with id fb51dc632eea870fa6b110d9994f576ffa506fe450c0ee6619a4cc188adec581
Nov 03 01:15:01.244000 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:01.243967    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbbcfc46-aa01-443c-a475-a757649f86ce" path="/var/lib/kubelet/pods/dbbcfc46-aa01-443c-a475-a757649f86ce/volumes"
Nov 03 01:15:01.523639 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:01.523590    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g" event={"ID":"8c26a270-d736-4678-9ff8-bfabcb06fe44","Type":"ContainerStarted","Data":"7755a3d36fb3b79a3c2c8e61268e22b8df8d1480a167d23b61da4af0988ea3e8"}
Nov 03 01:15:01.523839 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:01.523668    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g" event={"ID":"8c26a270-d736-4678-9ff8-bfabcb06fe44","Type":"ContainerStarted","Data":"fb419c0b12d3e078cbcf22db78f382e45d4f887855e1163e134cfcb448ca125d"}
Nov 03 01:15:01.526361 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:01.526335    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv" event={"ID":"c3c993ca-dfa2-422f-bb9b-21af3056b405","Type":"ContainerStarted","Data":"856e0fec90a803781985cefcdd868d08ea2aeb6592543cd109ae4ed5730d2631"}
Nov 03 01:15:01.526483 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:01.526381    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv" event={"ID":"c3c993ca-dfa2-422f-bb9b-21af3056b405","Type":"ContainerStarted","Data":"fb51dc632eea870fa6b110d9994f576ffa506fe450c0ee6619a4cc188adec581"}
Nov 03 01:15:01.560890 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:01.560844    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g" podStartSLOduration=1.560830756 podStartE2EDuration="1.560830756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:01.558123458 +0000 UTC m=+4184.437707500" watchObservedRunningTime="2026-11-03 01:15:01.560830756 +0000 UTC m=+4184.440414796"
Nov 03 01:15:01.594542 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:01.594488    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv" podStartSLOduration=1.5944740080000002 podStartE2EDuration="1.594474008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:01.592892275 +0000 UTC m=+4184.472476315" watchObservedRunningTime="2026-11-03 01:15:01.594474008 +0000 UTC m=+4184.474058074"
Nov 03 01:15:02.535557 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:02.535519    5012 generic.go:356] "Generic (PLEG): container finished" podID="8c26a270-d736-4678-9ff8-bfabcb06fe44" containerID="7755a3d36fb3b79a3c2c8e61268e22b8df8d1480a167d23b61da4af0988ea3e8" exitCode=60
Nov 03 01:15:02.536048 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:02.535609    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g" event={"ID":"8c26a270-d736-4678-9ff8-bfabcb06fe44","Type":"ContainerDied","Data":"7755a3d36fb3b79a3c2c8e61268e22b8df8d1480a167d23b61da4af0988ea3e8"}
Nov 03 01:15:02.536048 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:02.535812    5012 scope.go:122] "RemoveContainer" containerID="7755a3d36fb3b79a3c2c8e61268e22b8df8d1480a167d23b61da4af0988ea3e8"
Nov 03 01:15:02.537506 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:02.537488    5012 generic.go:356] "Generic (PLEG): container finished" podID="c3c993ca-dfa2-422f-bb9b-21af3056b405" containerID="856e0fec90a803781985cefcdd868d08ea2aeb6592543cd109ae4ed5730d2631" exitCode=60
Nov 03 01:15:02.537577 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:02.537517    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv" event={"ID":"c3c993ca-dfa2-422f-bb9b-21af3056b405","Type":"ContainerDied","Data":"856e0fec90a803781985cefcdd868d08ea2aeb6592543cd109ae4ed5730d2631"}
Nov 03 01:15:02.537763 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:02.537748    5012 scope.go:122] "RemoveContainer" containerID="856e0fec90a803781985cefcdd868d08ea2aeb6592543cd109ae4ed5730d2631"
Nov 03 01:15:03.546513 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:03.546481    5012 generic.go:356] "Generic (PLEG): container finished" podID="8c26a270-d736-4678-9ff8-bfabcb06fe44" containerID="da5dd933a2c347fdf973bd1bb946641ae294d8dcaace1cbace681b5a0e0d562a" exitCode=60
Nov 03 01:15:03.546864 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:03.546562    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g" event={"ID":"8c26a270-d736-4678-9ff8-bfabcb06fe44","Type":"ContainerDied","Data":"da5dd933a2c347fdf973bd1bb946641ae294d8dcaace1cbace681b5a0e0d562a"}
Nov 03 01:15:03.546864 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:03.546651    5012 scope.go:122] "RemoveContainer" containerID="7755a3d36fb3b79a3c2c8e61268e22b8df8d1480a167d23b61da4af0988ea3e8"
Nov 03 01:15:03.547017 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:03.546886    5012 scope.go:122] "RemoveContainer" containerID="da5dd933a2c347fdf973bd1bb946641ae294d8dcaace1cbace681b5a0e0d562a"
Nov 03 01:15:03.547234 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:15:03.547207    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g_odh-ai-gateway-infra(8c26a270-d736-4678-9ff8-bfabcb06fe44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g" podUID="8c26a270-d736-4678-9ff8-bfabcb06fe44"
Nov 03 01:15:03.548967 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:03.548944    5012 generic.go:356] "Generic (PLEG): container finished" podID="c3c993ca-dfa2-422f-bb9b-21af3056b405" containerID="03f2013eb9d44128d23cc75ac2c63aecef1854077bed5acd28364081c8b8f903" exitCode=60
Nov 03 01:15:03.549052 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:03.549031    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv" event={"ID":"c3c993ca-dfa2-422f-bb9b-21af3056b405","Type":"ContainerDied","Data":"03f2013eb9d44128d23cc75ac2c63aecef1854077bed5acd28364081c8b8f903"}
Nov 03 01:15:03.549285 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:03.549270    5012 scope.go:122] "RemoveContainer" containerID="03f2013eb9d44128d23cc75ac2c63aecef1854077bed5acd28364081c8b8f903"
Nov 03 01:15:03.549563 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:15:03.549541    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv_odh-ai-gateway-infra(c3c993ca-dfa2-422f-bb9b-21af3056b405)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv" podUID="c3c993ca-dfa2-422f-bb9b-21af3056b405"
Nov 03 01:15:03.629684 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:03.629656    5012 scope.go:122] "RemoveContainer" containerID="856e0fec90a803781985cefcdd868d08ea2aeb6592543cd109ae4ed5730d2631"
Nov 03 01:15:04.557263 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:04.557233    5012 scope.go:122] "RemoveContainer" containerID="03f2013eb9d44128d23cc75ac2c63aecef1854077bed5acd28364081c8b8f903"
Nov 03 01:15:04.557726 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:15:04.557536    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv_odh-ai-gateway-infra(c3c993ca-dfa2-422f-bb9b-21af3056b405)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv" podUID="c3c993ca-dfa2-422f-bb9b-21af3056b405"
Nov 03 01:15:04.559083 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:04.559061    5012 scope.go:122] "RemoveContainer" containerID="da5dd933a2c347fdf973bd1bb946641ae294d8dcaace1cbace681b5a0e0d562a"
Nov 03 01:15:04.559273 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:15:04.559255    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g_odh-ai-gateway-infra(8c26a270-d736-4678-9ff8-bfabcb06fe44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g" podUID="8c26a270-d736-4678-9ff8-bfabcb06fe44"
Nov 03 01:15:06.446732 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:06.446695    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957"]
Nov 03 01:15:07.362479 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:15:07.362436    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:15:11.447452 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:11.447411    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957" podUID="a325b3a8-8c15-40f2-8f63-47e0e15795a6" containerName="payload-pre-processing" containerID="cri-o://7552ec53e6e63a469bcce528b9153fff6ea1f77065c47d8711b22c6721a5464a" gracePeriod=25
Nov 03 01:15:11.612022 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:11.611986    5012 generic.go:356] "Generic (PLEG): container finished" podID="a325b3a8-8c15-40f2-8f63-47e0e15795a6" containerID="7552ec53e6e63a469bcce528b9153fff6ea1f77065c47d8711b22c6721a5464a" exitCode=0
Nov 03 01:15:11.612191 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:11.612069    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957" event={"ID":"a325b3a8-8c15-40f2-8f63-47e0e15795a6","Type":"ContainerDied","Data":"7552ec53e6e63a469bcce528b9153fff6ea1f77065c47d8711b22c6721a5464a"}
Nov 03 01:15:11.741831 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:11.741803    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957"
Nov 03 01:15:11.855402 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:11.855371    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a325b3a8-8c15-40f2-8f63-47e0e15795a6-kube-api-access-bshwm\" (UniqueName: \"kubernetes.io/projected/a325b3a8-8c15-40f2-8f63-47e0e15795a6-kube-api-access-bshwm\") pod \"a325b3a8-8c15-40f2-8f63-47e0e15795a6\" (UID: \"a325b3a8-8c15-40f2-8f63-47e0e15795a6\") "
Nov 03 01:15:11.855570 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:11.855509    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a325b3a8-8c15-40f2-8f63-47e0e15795a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a325b3a8-8c15-40f2-8f63-47e0e15795a6-plugins-config-volume\") pod \"a325b3a8-8c15-40f2-8f63-47e0e15795a6\" (UID: \"a325b3a8-8c15-40f2-8f63-47e0e15795a6\") "
Nov 03 01:15:11.855864 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:11.855833    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a325b3a8-8c15-40f2-8f63-47e0e15795a6-plugins-config-volume" pod "a325b3a8-8c15-40f2-8f63-47e0e15795a6" (UID: "a325b3a8-8c15-40f2-8f63-47e0e15795a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:11.857535 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:11.857505    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a325b3a8-8c15-40f2-8f63-47e0e15795a6-kube-api-access-bshwm" pod "a325b3a8-8c15-40f2-8f63-47e0e15795a6" (UID: "a325b3a8-8c15-40f2-8f63-47e0e15795a6"). InnerVolumeSpecName "kube-api-access-bshwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:11.956761 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:11.956736    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a325b3a8-8c15-40f2-8f63-47e0e15795a6-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:11.956859 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:11.956767    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bshwm\" (UniqueName: \"kubernetes.io/projected/a325b3a8-8c15-40f2-8f63-47e0e15795a6-kube-api-access-bshwm\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:12.622130 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:12.622094    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957"
Nov 03 01:15:12.622130 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:12.622118    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957" event={"ID":"a325b3a8-8c15-40f2-8f63-47e0e15795a6","Type":"ContainerDied","Data":"157c4a309d910a142fd1c0238d42507007ffe11d7b89167d14723c6c74398c2d"}
Nov 03 01:15:12.622730 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:12.622177    5012 scope.go:122] "RemoveContainer" containerID="7552ec53e6e63a469bcce528b9153fff6ea1f77065c47d8711b22c6721a5464a"
Nov 03 01:15:12.670224 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:12.670191    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957"]
Nov 03 01:15:12.679273 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:12.679242    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-4c61c8-9dc48565f-fh957"]
Nov 03 01:15:13.243633 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:13.243599    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a325b3a8-8c15-40f2-8f63-47e0e15795a6" path="/var/lib/kubelet/pods/a325b3a8-8c15-40f2-8f63-47e0e15795a6/volumes"
Nov 03 01:15:14.236513 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:14.236485    5012 scope.go:122] "RemoveContainer" containerID="da5dd933a2c347fdf973bd1bb946641ae294d8dcaace1cbace681b5a0e0d562a"
Nov 03 01:15:14.638988 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:14.638958    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g" event={"ID":"8c26a270-d736-4678-9ff8-bfabcb06fe44","Type":"ContainerStarted","Data":"78f914bd7e3ff500c736ef31b65d75d3252065ddad10448a729176ead5792c56"}
Nov 03 01:15:14.998740 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:14.998700    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main" containerID="cri-o://eb88e654aa294725147144e20cb6b7381ce6888f17999650f12202ea91590c90" gracePeriod=45
Nov 03 01:15:15.256472 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.256414    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 01:15:15.306028 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.305995    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-kube-api-access-92p8l\" (UniqueName: \"kubernetes.io/projected/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-kube-api-access-92p8l\") pod \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\" (UID: \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\") "
Nov 03 01:15:15.306190 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.306147    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-home\" (UniqueName: \"kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-home\") pod \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\" (UID: \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\") "
Nov 03 01:15:15.306257 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.306221    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-dshm\") pod \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\" (UID: \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\") "
Nov 03 01:15:15.306316 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.306257    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-tmp-dir\") pod \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\" (UID: \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\") "
Nov 03 01:15:15.306372 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.306313    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-model-cache\") pod \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\" (UID: \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\") "
Nov 03 01:15:15.306372 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.306364    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-tls-certs\" (UniqueName: \"kubernetes.io/secret/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-tls-certs\") pod \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\" (UID: \"c91595ba-7c00-44a8-b6d5-2bb7499b54b3\") "
Nov 03 01:15:15.306497 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.306465    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-home" pod "c91595ba-7c00-44a8-b6d5-2bb7499b54b3" (UID: "c91595ba-7c00-44a8-b6d5-2bb7499b54b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:15.306682 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.306657    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-tmp-dir" pod "c91595ba-7c00-44a8-b6d5-2bb7499b54b3" (UID: "c91595ba-7c00-44a8-b6d5-2bb7499b54b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:15.309831 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.306871    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-model-cache" pod "c91595ba-7c00-44a8-b6d5-2bb7499b54b3" (UID: "c91595ba-7c00-44a8-b6d5-2bb7499b54b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:15.309831 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.307076    5012 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-tmp-dir\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:15.309831 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.307100    5012 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-model-cache\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:15.309831 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.307112    5012 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-home\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:15.309831 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.309662    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-tls-certs" pod "c91595ba-7c00-44a8-b6d5-2bb7499b54b3" (UID: "c91595ba-7c00-44a8-b6d5-2bb7499b54b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:15:15.310627 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.310589    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-kube-api-access-92p8l" pod "c91595ba-7c00-44a8-b6d5-2bb7499b54b3" (UID: "c91595ba-7c00-44a8-b6d5-2bb7499b54b3"). InnerVolumeSpecName "kube-api-access-92p8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:15.311354 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.311330    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-dshm" pod "c91595ba-7c00-44a8-b6d5-2bb7499b54b3" (UID: "c91595ba-7c00-44a8-b6d5-2bb7499b54b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:15.408076 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.408039    5012 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-tls-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:15.408076 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.408069    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92p8l\" (UniqueName: \"kubernetes.io/projected/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-kube-api-access-92p8l\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:15.408076 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.408080    5012 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c91595ba-7c00-44a8-b6d5-2bb7499b54b3-dshm\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:15.650948 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.650880    5012 generic.go:356] "Generic (PLEG): container finished" podID="8c26a270-d736-4678-9ff8-bfabcb06fe44" containerID="78f914bd7e3ff500c736ef31b65d75d3252065ddad10448a729176ead5792c56" exitCode=60
Nov 03 01:15:15.651148 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.650965    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g" event={"ID":"8c26a270-d736-4678-9ff8-bfabcb06fe44","Type":"ContainerDied","Data":"78f914bd7e3ff500c736ef31b65d75d3252065ddad10448a729176ead5792c56"}
Nov 03 01:15:15.651148 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.651036    5012 scope.go:122] "RemoveContainer" containerID="da5dd933a2c347fdf973bd1bb946641ae294d8dcaace1cbace681b5a0e0d562a"
Nov 03 01:15:15.651518 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.651497    5012 scope.go:122] "RemoveContainer" containerID="78f914bd7e3ff500c736ef31b65d75d3252065ddad10448a729176ead5792c56"
Nov 03 01:15:15.651882 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:15:15.651849    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g_odh-ai-gateway-infra(8c26a270-d736-4678-9ff8-bfabcb06fe44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g" podUID="8c26a270-d736-4678-9ff8-bfabcb06fe44"
Nov 03 01:15:15.652647 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.652626    5012 generic.go:356] "Generic (PLEG): container finished" podID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerID="eb88e654aa294725147144e20cb6b7381ce6888f17999650f12202ea91590c90" exitCode=0
Nov 03 01:15:15.652836 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.652820    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" event={"ID":"c91595ba-7c00-44a8-b6d5-2bb7499b54b3","Type":"ContainerDied","Data":"eb88e654aa294725147144e20cb6b7381ce6888f17999650f12202ea91590c90"}
Nov 03 01:15:15.652942 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.652858    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm" event={"ID":"c91595ba-7c00-44a8-b6d5-2bb7499b54b3","Type":"ContainerDied","Data":"8842a340ed04503630b72fca661985d18608ac918abe1389c99e56e68ce5a681"}
Nov 03 01:15:15.653018 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.652946    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"
Nov 03 01:15:15.665419 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.665394    5012 scope.go:122] "RemoveContainer" containerID="eb88e654aa294725147144e20cb6b7381ce6888f17999650f12202ea91590c90"
Nov 03 01:15:15.674839 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.674816    5012 scope.go:122] "RemoveContainer" containerID="eb88e654aa294725147144e20cb6b7381ce6888f17999650f12202ea91590c90"
Nov 03 01:15:15.675190 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:15:15.675156    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb88e654aa294725147144e20cb6b7381ce6888f17999650f12202ea91590c90\": container with ID starting with eb88e654aa294725147144e20cb6b7381ce6888f17999650f12202ea91590c90 not found: ID does not exist" containerID="eb88e654aa294725147144e20cb6b7381ce6888f17999650f12202ea91590c90"
Nov 03 01:15:15.675290 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.675192    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb88e654aa294725147144e20cb6b7381ce6888f17999650f12202ea91590c90"} err="failed to get container status \"eb88e654aa294725147144e20cb6b7381ce6888f17999650f12202ea91590c90\": rpc error: code = NotFound desc = could not find container \"eb88e654aa294725147144e20cb6b7381ce6888f17999650f12202ea91590c90\": container with ID starting with eb88e654aa294725147144e20cb6b7381ce6888f17999650f12202ea91590c90 not found: ID does not exist"
Nov 03 01:15:15.690653 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.690629    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g"]
Nov 03 01:15:15.739435 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.739400    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"]
Nov 03 01:15:15.743229 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:15.743192    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-4c61c8/distinct-w2-4c61c8-kserve-5c96989bd-zqhpm"]
Nov 03 01:15:16.236105 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:16.236079    5012 scope.go:122] "RemoveContainer" containerID="03f2013eb9d44128d23cc75ac2c63aecef1854077bed5acd28364081c8b8f903"
Nov 03 01:15:17.091581 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:17.091544    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g"
Nov 03 01:15:17.154594 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:17.154555    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c26a270-d736-4678-9ff8-bfabcb06fe44-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c26a270-d736-4678-9ff8-bfabcb06fe44-maas-api-service-ca\") pod \"8c26a270-d736-4678-9ff8-bfabcb06fe44\" (UID: \"8c26a270-d736-4678-9ff8-bfabcb06fe44\") "
Nov 03 01:15:17.154831 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:17.154670    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c26a270-d736-4678-9ff8-bfabcb06fe44-kube-api-access-plnb8\" (UniqueName: \"kubernetes.io/projected/8c26a270-d736-4678-9ff8-bfabcb06fe44-kube-api-access-plnb8\") pod \"8c26a270-d736-4678-9ff8-bfabcb06fe44\" (UID: \"8c26a270-d736-4678-9ff8-bfabcb06fe44\") "
Nov 03 01:15:17.155137 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:17.155108    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c26a270-d736-4678-9ff8-bfabcb06fe44-maas-api-service-ca" pod "8c26a270-d736-4678-9ff8-bfabcb06fe44" (UID: "8c26a270-d736-4678-9ff8-bfabcb06fe44"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:17.158067 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:17.158035    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c26a270-d736-4678-9ff8-bfabcb06fe44-kube-api-access-plnb8" pod "8c26a270-d736-4678-9ff8-bfabcb06fe44" (UID: "8c26a270-d736-4678-9ff8-bfabcb06fe44"). InnerVolumeSpecName "kube-api-access-plnb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:17.256587 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:17.256532    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c26a270-d736-4678-9ff8-bfabcb06fe44-maas-api-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:17.256587 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:17.256562    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plnb8\" (UniqueName: \"kubernetes.io/projected/8c26a270-d736-4678-9ff8-bfabcb06fe44-kube-api-access-plnb8\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:17.261162 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:17.261129    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" path="/var/lib/kubelet/pods/c91595ba-7c00-44a8-b6d5-2bb7499b54b3/volumes"
Nov 03 01:15:17.711931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:17.710935    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g" event={"ID":"8c26a270-d736-4678-9ff8-bfabcb06fe44","Type":"ContainerDied","Data":"fb419c0b12d3e078cbcf22db78f382e45d4f887855e1163e134cfcb448ca125d"}
Nov 03 01:15:17.711931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:17.711012    5012 scope.go:122] "RemoveContainer" containerID="78f914bd7e3ff500c736ef31b65d75d3252065ddad10448a729176ead5792c56"
Nov 03 01:15:17.711931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:17.711133    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g"
Nov 03 01:15:17.735373 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:17.732618    5012 generic.go:356] "Generic (PLEG): container finished" podID="c3c993ca-dfa2-422f-bb9b-21af3056b405" containerID="38be176a09814bfe840ac0311f7a3abf9e04c609b3b1c16d312edd0a72655167" exitCode=60
Nov 03 01:15:17.735373 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:17.732744    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv" event={"ID":"c3c993ca-dfa2-422f-bb9b-21af3056b405","Type":"ContainerDied","Data":"38be176a09814bfe840ac0311f7a3abf9e04c609b3b1c16d312edd0a72655167"}
Nov 03 01:15:17.735373 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:17.733263    5012 scope.go:122] "RemoveContainer" containerID="38be176a09814bfe840ac0311f7a3abf9e04c609b3b1c16d312edd0a72655167"
Nov 03 01:15:17.735373 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:15:17.734051    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv_odh-ai-gateway-infra(c3c993ca-dfa2-422f-bb9b-21af3056b405)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv" podUID="c3c993ca-dfa2-422f-bb9b-21af3056b405"
Nov 03 01:15:17.747198 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:17.743793    5012 scope.go:122] "RemoveContainer" containerID="03f2013eb9d44128d23cc75ac2c63aecef1854077bed5acd28364081c8b8f903"
Nov 03 01:15:17.791013 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:17.790971    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g"]
Nov 03 01:15:17.877009 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:17.876965    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894475-vzj9g"]
Nov 03 01:15:18.616589 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:18.616552    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 01:15:18.621569 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:18.621545    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 01:15:18.626007 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:18.625970    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 01:15:18.630834 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:18.630809    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 01:15:18.818359 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:18.818324    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv"]
Nov 03 01:15:18.986769 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:18.986742    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv"
Nov 03 01:15:19.087102 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:19.087063    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3c993ca-dfa2-422f-bb9b-21af3056b405-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c993ca-dfa2-422f-bb9b-21af3056b405-maas-api-service-ca\") pod \"c3c993ca-dfa2-422f-bb9b-21af3056b405\" (UID: \"c3c993ca-dfa2-422f-bb9b-21af3056b405\") "
Nov 03 01:15:19.087266 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:19.087253    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3c993ca-dfa2-422f-bb9b-21af3056b405-kube-api-access-mfcnw\" (UniqueName: \"kubernetes.io/projected/c3c993ca-dfa2-422f-bb9b-21af3056b405-kube-api-access-mfcnw\") pod \"c3c993ca-dfa2-422f-bb9b-21af3056b405\" (UID: \"c3c993ca-dfa2-422f-bb9b-21af3056b405\") "
Nov 03 01:15:19.087434 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:19.087409    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c993ca-dfa2-422f-bb9b-21af3056b405-maas-api-service-ca" pod "c3c993ca-dfa2-422f-bb9b-21af3056b405" (UID: "c3c993ca-dfa2-422f-bb9b-21af3056b405"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:19.087571 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:19.087555    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c993ca-dfa2-422f-bb9b-21af3056b405-maas-api-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:19.089368 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:19.089336    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c993ca-dfa2-422f-bb9b-21af3056b405-kube-api-access-mfcnw" pod "c3c993ca-dfa2-422f-bb9b-21af3056b405" (UID: "c3c993ca-dfa2-422f-bb9b-21af3056b405"). InnerVolumeSpecName "kube-api-access-mfcnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:19.188401 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:19.188323    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfcnw\" (UniqueName: \"kubernetes.io/projected/c3c993ca-dfa2-422f-bb9b-21af3056b405-kube-api-access-mfcnw\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:19.244370 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:19.244336    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c26a270-d736-4678-9ff8-bfabcb06fe44" path="/var/lib/kubelet/pods/8c26a270-d736-4678-9ff8-bfabcb06fe44/volumes"
Nov 03 01:15:19.823084 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:19.823035    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv" event={"ID":"c3c993ca-dfa2-422f-bb9b-21af3056b405","Type":"ContainerDied","Data":"fb51dc632eea870fa6b110d9994f576ffa506fe450c0ee6619a4cc188adec581"}
Nov 03 01:15:19.824417 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:19.823115    5012 scope.go:122] "RemoveContainer" containerID="38be176a09814bfe840ac0311f7a3abf9e04c609b3b1c16d312edd0a72655167"
Nov 03 01:15:19.824417 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:19.823274    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv"
Nov 03 01:15:19.873302 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:19.873272    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv"]
Nov 03 01:15:19.883640 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:19.883612    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894475-bq4vv"]
Nov 03 01:15:21.243307 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:21.243277    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3c993ca-dfa2-422f-bb9b-21af3056b405" path="/var/lib/kubelet/pods/c3c993ca-dfa2-422f-bb9b-21af3056b405/volumes"
Nov 03 01:15:37.364761 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:15:37.364723    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:15:37.648576 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:37.648478    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c"]
Nov 03 01:15:37.649061 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:37.649021    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c" podUID="1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a" containerName="maas-api" containerID="cri-o://1af5310ce3f0ee022062254ec75a540dfc508f048a87dda724439d26956a55fe" gracePeriod=30
Nov 03 01:15:37.962624 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:37.962595    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c"
Nov 03 01:15:37.977333 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:37.977298    5012 generic.go:356] "Generic (PLEG): container finished" podID="1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a" containerID="1af5310ce3f0ee022062254ec75a540dfc508f048a87dda724439d26956a55fe" exitCode=0
Nov 03 01:15:37.977431 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:37.977351    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c"
Nov 03 01:15:37.977534 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:37.977503    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c" event={"ID":"1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a","Type":"ContainerDied","Data":"1af5310ce3f0ee022062254ec75a540dfc508f048a87dda724439d26956a55fe"}
Nov 03 01:15:37.977667 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:37.977560    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c" event={"ID":"1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a","Type":"ContainerDied","Data":"33018606f06f08a0b78ecc0d53528f65601e0d6b6ee31a49ab52db5f12691bb0"}
Nov 03 01:15:37.977667 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:37.977585    5012 scope.go:122] "RemoveContainer" containerID="1af5310ce3f0ee022062254ec75a540dfc508f048a87dda724439d26956a55fe"
Nov 03 01:15:37.987691 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:37.987668    5012 scope.go:122] "RemoveContainer" containerID="1af5310ce3f0ee022062254ec75a540dfc508f048a87dda724439d26956a55fe"
Nov 03 01:15:37.988039 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:15:37.987994    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af5310ce3f0ee022062254ec75a540dfc508f048a87dda724439d26956a55fe\": container with ID starting with 1af5310ce3f0ee022062254ec75a540dfc508f048a87dda724439d26956a55fe not found: ID does not exist" containerID="1af5310ce3f0ee022062254ec75a540dfc508f048a87dda724439d26956a55fe"
Nov 03 01:15:37.988113 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:37.988049    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af5310ce3f0ee022062254ec75a540dfc508f048a87dda724439d26956a55fe"} err="failed to get container status \"1af5310ce3f0ee022062254ec75a540dfc508f048a87dda724439d26956a55fe\": rpc error: code = NotFound desc = could not find container \"1af5310ce3f0ee022062254ec75a540dfc508f048a87dda724439d26956a55fe\": container with ID starting with 1af5310ce3f0ee022062254ec75a540dfc508f048a87dda724439d26956a55fe not found: ID does not exist"
Nov 03 01:15:38.046113 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:38.046085    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-openshift-service-ca\") pod \"1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a\" (UID: \"1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a\") "
Nov 03 01:15:38.046287 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:38.046138    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-kube-api-access-29m4k\" (UniqueName: \"kubernetes.io/projected/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-kube-api-access-29m4k\") pod \"1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a\" (UID: \"1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a\") "
Nov 03 01:15:38.046287 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:38.046183    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-metrics-certs\") pod \"1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a\" (UID: \"1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a\") "
Nov 03 01:15:38.046421 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:38.046398    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-maas-api-tls\") pod \"1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a\" (UID: \"1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a\") "
Nov 03 01:15:38.046488 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:38.046443    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-openshift-service-ca" pod "1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a" (UID: "1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:38.046832 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:38.046812    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:38.048445 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:38.048410    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-metrics-certs" pod "1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a" (UID: "1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:15:38.048564 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:38.048494    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-maas-api-tls" pod "1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a" (UID: "1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:15:38.048630 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:38.048573    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-kube-api-access-29m4k" pod "1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a" (UID: "1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a"). InnerVolumeSpecName "kube-api-access-29m4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:38.160934 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:38.155775    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:38.160934 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:38.155822    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29m4k\" (UniqueName: \"kubernetes.io/projected/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-kube-api-access-29m4k\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:38.160934 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:38.155839    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:38.320476 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:38.320444    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c"]
Nov 03 01:15:38.330903 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:38.330873    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-850433-84b54b5bbc-q5k9c"]
Nov 03 01:15:39.243215 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:39.243175    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a" path="/var/lib/kubelet/pods/1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a/volumes"
Nov 03 01:15:42.839197 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.839163    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"]
Nov 03 01:15:42.839740 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.839712    5012 cpu_manager.go:414] "containerMap: removing container" podUID="1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a" containerName="maas-api"
Nov 03 01:15:42.839740 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.839730    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a" containerName="maas-api"
Nov 03 01:15:42.839740 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.839741    5012 cpu_manager.go:414] "containerMap: removing container" podUID="8c26a270-d736-4678-9ff8-bfabcb06fe44" containerName="cleanup"
Nov 03 01:15:42.839994 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.839747    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c26a270-d736-4678-9ff8-bfabcb06fe44" containerName="cleanup"
Nov 03 01:15:42.839994 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.839756    5012 cpu_manager.go:414] "containerMap: removing container" podUID="c3c993ca-dfa2-422f-bb9b-21af3056b405" containerName="cleanup"
Nov 03 01:15:42.839994 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.839763    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c993ca-dfa2-422f-bb9b-21af3056b405" containerName="cleanup"
Nov 03 01:15:42.839994 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.839783    5012 cpu_manager.go:414] "containerMap: removing container" podUID="a325b3a8-8c15-40f2-8f63-47e0e15795a6" containerName="payload-pre-processing"
Nov 03 01:15:42.839994 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.839788    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a325b3a8-8c15-40f2-8f63-47e0e15795a6" containerName="payload-pre-processing"
Nov 03 01:15:42.839994 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.839802    5012 cpu_manager.go:414] "containerMap: removing container" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main"
Nov 03 01:15:42.839994 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.839806    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main"
Nov 03 01:15:42.839994 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.839825    5012 cpu_manager.go:414] "containerMap: removing container" podUID="8c26a270-d736-4678-9ff8-bfabcb06fe44" containerName="cleanup"
Nov 03 01:15:42.839994 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.839830    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c26a270-d736-4678-9ff8-bfabcb06fe44" containerName="cleanup"
Nov 03 01:15:42.839994 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.839839    5012 cpu_manager.go:414] "containerMap: removing container" podUID="8c26a270-d736-4678-9ff8-bfabcb06fe44" containerName="cleanup"
Nov 03 01:15:42.839994 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.839845    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c26a270-d736-4678-9ff8-bfabcb06fe44" containerName="cleanup"
Nov 03 01:15:42.839994 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.839859    5012 cpu_manager.go:414] "containerMap: removing container" podUID="c3c993ca-dfa2-422f-bb9b-21af3056b405" containerName="cleanup"
Nov 03 01:15:42.839994 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.839864    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c993ca-dfa2-422f-bb9b-21af3056b405" containerName="cleanup"
Nov 03 01:15:42.839994 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.839967    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b1773fa-7dcd-4af7-8c29-0ba8fd2a166a" containerName="maas-api"
Nov 03 01:15:42.839994 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.839976    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c26a270-d736-4678-9ff8-bfabcb06fe44" containerName="cleanup"
Nov 03 01:15:42.839994 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.839982    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c993ca-dfa2-422f-bb9b-21af3056b405" containerName="cleanup"
Nov 03 01:15:42.839994 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.839989    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="a325b3a8-8c15-40f2-8f63-47e0e15795a6" containerName="payload-pre-processing"
Nov 03 01:15:42.839994 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.839999    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="c91595ba-7c00-44a8-b6d5-2bb7499b54b3" containerName="main"
Nov 03 01:15:42.840837 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.840012    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c26a270-d736-4678-9ff8-bfabcb06fe44" containerName="cleanup"
Nov 03 01:15:42.840837 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.840023    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c993ca-dfa2-422f-bb9b-21af3056b405" containerName="cleanup"
Nov 03 01:15:42.840837 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.840034    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c993ca-dfa2-422f-bb9b-21af3056b405" containerName="cleanup"
Nov 03 01:15:42.845660 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.845635    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:42.873575 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.873549    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-4c61c8-openshift-default-dockercfg-sclgr\""
Nov 03 01:15:42.883545 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:42.883501    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"]
Nov 03 01:15:43.014931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.010783    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.014931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.010859    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.014931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.010946    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.014931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.010988    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d206e64-ac15-4522-846a-e6c10863f235-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.014931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.011017    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.014931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.011138    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d206e64-ac15-4522-846a-e6c10863f235-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.014931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.011285    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.014931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.011338    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkv8v\" (UniqueName: \"kubernetes.io/projected/0d206e64-ac15-4522-846a-e6c10863f235-kube-api-access-gkv8v\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.014931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.011493    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d206e64-ac15-4522-846a-e6c10863f235-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.112537 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.112507    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d206e64-ac15-4522-846a-e6c10863f235-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.112747 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.112548    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.112747 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.112581    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d206e64-ac15-4522-846a-e6c10863f235-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.112747 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.112631    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.112747 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.112661    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkv8v\" (UniqueName: \"kubernetes.io/projected/0d206e64-ac15-4522-846a-e6c10863f235-kube-api-access-gkv8v\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.112747 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.112736    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d206e64-ac15-4522-846a-e6c10863f235-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.113089 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.112768    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.113089 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.112808    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.113089 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.112865    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.113089 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.113057    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.113089 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.113074    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.113615 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.113260    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.113615 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.113319    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.113615 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.113502    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d206e64-ac15-4522-846a-e6c10863f235-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.115199 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.115181    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d206e64-ac15-4522-846a-e6c10863f235-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.115338 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.115319    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.129710 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.129680    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d206e64-ac15-4522-846a-e6c10863f235-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.129813 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.129764    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkv8v\" (UniqueName: \"kubernetes.io/projected/0d206e64-ac15-4522-846a-e6c10863f235-kube-api-access-gkv8v\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.169385 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.169360    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:43.483675 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.483644    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"]
Nov 03 01:15:43.486569 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:15:43.485875    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d206e64_ac15_4522_846a_e6c10863f235.slice/crio-b9f50e10864d8cb561db1eb787456f0cbf191a6b70f75f8206b360ea948cbc96 WatchSource:0}: Error finding container b9f50e10864d8cb561db1eb787456f0cbf191a6b70f75f8206b360ea948cbc96: Status 404 returned error can't find the container with id b9f50e10864d8cb561db1eb787456f0cbf191a6b70f75f8206b360ea948cbc96
Nov 03 01:15:43.490427 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.490386    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:15:43.491062 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.491032    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:15:43.491260 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:43.491229    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:15:44.047894 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:44.047854    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7" event={"ID":"0d206e64-ac15-4522-846a-e6c10863f235","Type":"ContainerStarted","Data":"2f9d5348caf95baef4f58dc9425366019dadef31d1c0e8f395054f071d918890"}
Nov 03 01:15:44.048316 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:44.047965    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7" event={"ID":"0d206e64-ac15-4522-846a-e6c10863f235","Type":"ContainerStarted","Data":"b9f50e10864d8cb561db1eb787456f0cbf191a6b70f75f8206b360ea948cbc96"}
Nov 03 01:15:44.175214 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:44.172448    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:44.176076 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:44.176039    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Nov 03 01:15:44.176325 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:44.176293    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7" podUID="0d206e64-ac15-4522-846a-e6c10863f235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Nov 03 01:15:45.170006 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:45.169970    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Nov 03 01:15:45.170413 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:45.170035    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7" podUID="0d206e64-ac15-4522-846a-e6c10863f235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Nov 03 01:15:46.170516 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:46.170480    5012 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Nov 03 01:15:46.170950 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:46.170535    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7" podUID="0d206e64-ac15-4522-846a-e6c10863f235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Nov 03 01:15:47.132958 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:47.132873    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7" podStartSLOduration=5.132852847 podStartE2EDuration="5.132852847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:44.103245011 +0000 UTC m=+4226.982829055" watchObservedRunningTime="2026-11-03 01:15:47.132852847 +0000 UTC m=+4230.012436892"
Nov 03 01:15:47.137540 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:47.137515    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"]
Nov 03 01:15:47.137880 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:47.137851    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7" podUID="0d206e64-ac15-4522-846a-e6c10863f235" containerName="istio-proxy" containerID="cri-o://2f9d5348caf95baef4f58dc9425366019dadef31d1c0e8f395054f071d918890" gracePeriod=30
Nov 03 01:15:56.105975 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.105950    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:56.142017 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.141992    5012 generic.go:356] "Generic (PLEG): container finished" podID="0d206e64-ac15-4522-846a-e6c10863f235" containerID="2f9d5348caf95baef4f58dc9425366019dadef31d1c0e8f395054f071d918890" exitCode=0
Nov 03 01:15:56.142114 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.142034    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"
Nov 03 01:15:56.142114 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.142055    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7" event={"ID":"0d206e64-ac15-4522-846a-e6c10863f235","Type":"ContainerDied","Data":"2f9d5348caf95baef4f58dc9425366019dadef31d1c0e8f395054f071d918890"}
Nov 03 01:15:56.142114 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.142097    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7" event={"ID":"0d206e64-ac15-4522-846a-e6c10863f235","Type":"ContainerDied","Data":"b9f50e10864d8cb561db1eb787456f0cbf191a6b70f75f8206b360ea948cbc96"}
Nov 03 01:15:56.142249 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.142119    5012 scope.go:122] "RemoveContainer" containerID="2f9d5348caf95baef4f58dc9425366019dadef31d1c0e8f395054f071d918890"
Nov 03 01:15:56.152301 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.152276    5012 scope.go:122] "RemoveContainer" containerID="2f9d5348caf95baef4f58dc9425366019dadef31d1c0e8f395054f071d918890"
Nov 03 01:15:56.152558 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:15:56.152539    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f9d5348caf95baef4f58dc9425366019dadef31d1c0e8f395054f071d918890\": container with ID starting with 2f9d5348caf95baef4f58dc9425366019dadef31d1c0e8f395054f071d918890 not found: ID does not exist" containerID="2f9d5348caf95baef4f58dc9425366019dadef31d1c0e8f395054f071d918890"
Nov 03 01:15:56.152598 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.152570    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f9d5348caf95baef4f58dc9425366019dadef31d1c0e8f395054f071d918890"} err="failed to get container status \"2f9d5348caf95baef4f58dc9425366019dadef31d1c0e8f395054f071d918890\": rpc error: code = NotFound desc = could not find container \"2f9d5348caf95baef4f58dc9425366019dadef31d1c0e8f395054f071d918890\": container with ID starting with 2f9d5348caf95baef4f58dc9425366019dadef31d1c0e8f395054f071d918890 not found: ID does not exist"
Nov 03 01:15:56.206147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.206112    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d206e64-ac15-4522-846a-e6c10863f235-istio-token\" (UniqueName: \"kubernetes.io/projected/0d206e64-ac15-4522-846a-e6c10863f235-istio-token\") pod \"0d206e64-ac15-4522-846a-e6c10863f235\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") "
Nov 03 01:15:56.206367 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.206180    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-workload-certs\") pod \"0d206e64-ac15-4522-846a-e6c10863f235\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") "
Nov 03 01:15:56.206367 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.206283    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d206e64-ac15-4522-846a-e6c10863f235-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d206e64-ac15-4522-846a-e6c10863f235-istiod-ca-cert\") pod \"0d206e64-ac15-4522-846a-e6c10863f235\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") "
Nov 03 01:15:56.206367 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.206310    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-credential-socket\") pod \"0d206e64-ac15-4522-846a-e6c10863f235\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") "
Nov 03 01:15:56.206558 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.206371    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d206e64-ac15-4522-846a-e6c10863f235-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d206e64-ac15-4522-846a-e6c10863f235-istio-podinfo\") pod \"0d206e64-ac15-4522-846a-e6c10863f235\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") "
Nov 03 01:15:56.206558 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.206410    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-istio-envoy\") pod \"0d206e64-ac15-4522-846a-e6c10863f235\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") "
Nov 03 01:15:56.206558 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.206453    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d206e64-ac15-4522-846a-e6c10863f235-kube-api-access-gkv8v\" (UniqueName: \"kubernetes.io/projected/0d206e64-ac15-4522-846a-e6c10863f235-kube-api-access-gkv8v\") pod \"0d206e64-ac15-4522-846a-e6c10863f235\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") "
Nov 03 01:15:56.207218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.207188    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d206e64-ac15-4522-846a-e6c10863f235-istiod-ca-cert" pod "0d206e64-ac15-4522-846a-e6c10863f235" (UID: "0d206e64-ac15-4522-846a-e6c10863f235"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:56.207506 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.207363    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-credential-socket" pod "0d206e64-ac15-4522-846a-e6c10863f235" (UID: "0d206e64-ac15-4522-846a-e6c10863f235"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:56.207675 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.207380    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-workload-certs" pod "0d206e64-ac15-4522-846a-e6c10863f235" (UID: "0d206e64-ac15-4522-846a-e6c10863f235"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:56.209696 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.209665    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d206e64-ac15-4522-846a-e6c10863f235-istio-podinfo" pod "0d206e64-ac15-4522-846a-e6c10863f235" (UID: "0d206e64-ac15-4522-846a-e6c10863f235"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:15:56.211103 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.211075    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d206e64-ac15-4522-846a-e6c10863f235-istio-token" pod "0d206e64-ac15-4522-846a-e6c10863f235" (UID: "0d206e64-ac15-4522-846a-e6c10863f235"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:56.213778 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.213752    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-istio-envoy" pod "0d206e64-ac15-4522-846a-e6c10863f235" (UID: "0d206e64-ac15-4522-846a-e6c10863f235"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:56.213899 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.213876    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d206e64-ac15-4522-846a-e6c10863f235-kube-api-access-gkv8v" pod "0d206e64-ac15-4522-846a-e6c10863f235" (UID: "0d206e64-ac15-4522-846a-e6c10863f235"). InnerVolumeSpecName "kube-api-access-gkv8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:56.307563 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.307524    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-workload-socket\") pod \"0d206e64-ac15-4522-846a-e6c10863f235\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") "
Nov 03 01:15:56.307563 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.307562    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-istio-data\") pod \"0d206e64-ac15-4522-846a-e6c10863f235\" (UID: \"0d206e64-ac15-4522-846a-e6c10863f235\") "
Nov 03 01:15:56.307801 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.307766    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-workload-socket" pod "0d206e64-ac15-4522-846a-e6c10863f235" (UID: "0d206e64-ac15-4522-846a-e6c10863f235"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:56.307883 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.307863    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-istio-data" pod "0d206e64-ac15-4522-846a-e6c10863f235" (UID: "0d206e64-ac15-4522-846a-e6c10863f235"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:56.308006 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.307990    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d206e64-ac15-4522-846a-e6c10863f235-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:56.308082 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.308008    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:56.308082 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.308018    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d206e64-ac15-4522-846a-e6c10863f235-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:56.308082 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.308026    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:56.308082 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.308035    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d206e64-ac15-4522-846a-e6c10863f235-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:56.308082 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.308043    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:56.308082 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.308051    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkv8v\" (UniqueName: \"kubernetes.io/projected/0d206e64-ac15-4522-846a-e6c10863f235-kube-api-access-gkv8v\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:56.308082 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.308061    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:56.308082 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.308070    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d206e64-ac15-4522-846a-e6c10863f235-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:15:56.485341 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.485315    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"]
Nov 03 01:15:56.495057 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:56.495026    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-x98m7"]
Nov 03 01:15:57.243124 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:15:57.243093    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d206e64-ac15-4522-846a-e6c10863f235" path="/var/lib/kubelet/pods/0d206e64-ac15-4522-846a-e6c10863f235/volumes"
Nov 03 01:16:01.328871 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:01.328830    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7"]
Nov 03 01:16:01.329366 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:01.329103    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7" podUID="26130d50-27b4-4de9-8ec5-03af2ae23cbc" containerName="maas-api" containerID="cri-o://96a5ba0efcca8045432e5c12080bff3cc5e22c3767fb13d90654ae388a450d06" gracePeriod=30
Nov 03 01:16:01.639595 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:01.639557    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7"
Nov 03 01:16:01.663590 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:01.663562    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26130d50-27b4-4de9-8ec5-03af2ae23cbc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26130d50-27b4-4de9-8ec5-03af2ae23cbc-maas-api-tls\") pod \"26130d50-27b4-4de9-8ec5-03af2ae23cbc\" (UID: \"26130d50-27b4-4de9-8ec5-03af2ae23cbc\") "
Nov 03 01:16:01.663735 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:01.663602    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26130d50-27b4-4de9-8ec5-03af2ae23cbc-kube-api-access-lrlr8\" (UniqueName: \"kubernetes.io/projected/26130d50-27b4-4de9-8ec5-03af2ae23cbc-kube-api-access-lrlr8\") pod \"26130d50-27b4-4de9-8ec5-03af2ae23cbc\" (UID: \"26130d50-27b4-4de9-8ec5-03af2ae23cbc\") "
Nov 03 01:16:01.663735 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:01.663663    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26130d50-27b4-4de9-8ec5-03af2ae23cbc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26130d50-27b4-4de9-8ec5-03af2ae23cbc-openshift-service-ca\") pod \"26130d50-27b4-4de9-8ec5-03af2ae23cbc\" (UID: \"26130d50-27b4-4de9-8ec5-03af2ae23cbc\") "
Nov 03 01:16:01.663876 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:01.663857    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26130d50-27b4-4de9-8ec5-03af2ae23cbc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/26130d50-27b4-4de9-8ec5-03af2ae23cbc-metrics-certs\") pod \"26130d50-27b4-4de9-8ec5-03af2ae23cbc\" (UID: \"26130d50-27b4-4de9-8ec5-03af2ae23cbc\") "
Nov 03 01:16:01.664148 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:01.664118    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26130d50-27b4-4de9-8ec5-03af2ae23cbc-openshift-service-ca" pod "26130d50-27b4-4de9-8ec5-03af2ae23cbc" (UID: "26130d50-27b4-4de9-8ec5-03af2ae23cbc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:16:01.664352 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:01.664328    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26130d50-27b4-4de9-8ec5-03af2ae23cbc-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:01.665724 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:01.665701    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26130d50-27b4-4de9-8ec5-03af2ae23cbc-maas-api-tls" pod "26130d50-27b4-4de9-8ec5-03af2ae23cbc" (UID: "26130d50-27b4-4de9-8ec5-03af2ae23cbc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:16:01.665925 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:01.665889    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26130d50-27b4-4de9-8ec5-03af2ae23cbc-metrics-certs" pod "26130d50-27b4-4de9-8ec5-03af2ae23cbc" (UID: "26130d50-27b4-4de9-8ec5-03af2ae23cbc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:16:01.666028 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:01.665942    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26130d50-27b4-4de9-8ec5-03af2ae23cbc-kube-api-access-lrlr8" pod "26130d50-27b4-4de9-8ec5-03af2ae23cbc" (UID: "26130d50-27b4-4de9-8ec5-03af2ae23cbc"). InnerVolumeSpecName "kube-api-access-lrlr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:01.765219 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:01.765191    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26130d50-27b4-4de9-8ec5-03af2ae23cbc-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:01.765219 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:01.765216    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26130d50-27b4-4de9-8ec5-03af2ae23cbc-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:01.765379 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:01.765226    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrlr8\" (UniqueName: \"kubernetes.io/projected/26130d50-27b4-4de9-8ec5-03af2ae23cbc-kube-api-access-lrlr8\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:02.186373 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:02.186338    5012 generic.go:356] "Generic (PLEG): container finished" podID="26130d50-27b4-4de9-8ec5-03af2ae23cbc" containerID="96a5ba0efcca8045432e5c12080bff3cc5e22c3767fb13d90654ae388a450d06" exitCode=0
Nov 03 01:16:02.186545 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:02.186387    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7"
Nov 03 01:16:02.186545 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:02.186422    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7" event={"ID":"26130d50-27b4-4de9-8ec5-03af2ae23cbc","Type":"ContainerDied","Data":"96a5ba0efcca8045432e5c12080bff3cc5e22c3767fb13d90654ae388a450d06"}
Nov 03 01:16:02.186545 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:02.186486    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7" event={"ID":"26130d50-27b4-4de9-8ec5-03af2ae23cbc","Type":"ContainerDied","Data":"d9f1def129ac9cb565a0c103998d59b118c4ebfa5d6d0c5ffbabdc27e535c9bc"}
Nov 03 01:16:02.186545 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:02.186510    5012 scope.go:122] "RemoveContainer" containerID="96a5ba0efcca8045432e5c12080bff3cc5e22c3767fb13d90654ae388a450d06"
Nov 03 01:16:02.197458 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:02.197440    5012 scope.go:122] "RemoveContainer" containerID="96a5ba0efcca8045432e5c12080bff3cc5e22c3767fb13d90654ae388a450d06"
Nov 03 01:16:02.197826 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:16:02.197807    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96a5ba0efcca8045432e5c12080bff3cc5e22c3767fb13d90654ae388a450d06\": container with ID starting with 96a5ba0efcca8045432e5c12080bff3cc5e22c3767fb13d90654ae388a450d06 not found: ID does not exist" containerID="96a5ba0efcca8045432e5c12080bff3cc5e22c3767fb13d90654ae388a450d06"
Nov 03 01:16:02.197889 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:02.197833    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96a5ba0efcca8045432e5c12080bff3cc5e22c3767fb13d90654ae388a450d06"} err="failed to get container status \"96a5ba0efcca8045432e5c12080bff3cc5e22c3767fb13d90654ae388a450d06\": rpc error: code = NotFound desc = could not find container \"96a5ba0efcca8045432e5c12080bff3cc5e22c3767fb13d90654ae388a450d06\": container with ID starting with 96a5ba0efcca8045432e5c12080bff3cc5e22c3767fb13d90654ae388a450d06 not found: ID does not exist"
Nov 03 01:16:02.234855 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:02.234829    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7"]
Nov 03 01:16:02.244807 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:02.244775    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ac338d-58856dbcc9-5jbr7"]
Nov 03 01:16:03.242834 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:03.242804    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26130d50-27b4-4de9-8ec5-03af2ae23cbc" path="/var/lib/kubelet/pods/26130d50-27b4-4de9-8ec5-03af2ae23cbc/volumes"
Nov 03 01:16:07.363404 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:16:07.363368    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:16:10.098772 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:10.098739    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2"]
Nov 03 01:16:15.099581 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:15.099532    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2" podUID="a47c095a-1345-4bb2-a1db-b718dcf7261f" containerName="payload-pre-processing" containerID="cri-o://00f2ad9e5dd2396b090eb3259b66a189c64310f268b23da2d70b0f6549dd4e02" gracePeriod=25
Nov 03 01:16:15.276511 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:15.276466    5012 generic.go:356] "Generic (PLEG): container finished" podID="a47c095a-1345-4bb2-a1db-b718dcf7261f" containerID="00f2ad9e5dd2396b090eb3259b66a189c64310f268b23da2d70b0f6549dd4e02" exitCode=0
Nov 03 01:16:15.276656 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:15.276513    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2" event={"ID":"a47c095a-1345-4bb2-a1db-b718dcf7261f","Type":"ContainerDied","Data":"00f2ad9e5dd2396b090eb3259b66a189c64310f268b23da2d70b0f6549dd4e02"}
Nov 03 01:16:15.357892 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:15.357829    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2"
Nov 03 01:16:15.471161 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:15.471131    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a47c095a-1345-4bb2-a1db-b718dcf7261f-kube-api-access-lwc9x\" (UniqueName: \"kubernetes.io/projected/a47c095a-1345-4bb2-a1db-b718dcf7261f-kube-api-access-lwc9x\") pod \"a47c095a-1345-4bb2-a1db-b718dcf7261f\" (UID: \"a47c095a-1345-4bb2-a1db-b718dcf7261f\") "
Nov 03 01:16:15.471319 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:15.471240    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a47c095a-1345-4bb2-a1db-b718dcf7261f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47c095a-1345-4bb2-a1db-b718dcf7261f-plugins-config-volume\") pod \"a47c095a-1345-4bb2-a1db-b718dcf7261f\" (UID: \"a47c095a-1345-4bb2-a1db-b718dcf7261f\") "
Nov 03 01:16:15.471604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:15.471582    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a47c095a-1345-4bb2-a1db-b718dcf7261f-plugins-config-volume" pod "a47c095a-1345-4bb2-a1db-b718dcf7261f" (UID: "a47c095a-1345-4bb2-a1db-b718dcf7261f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:16:15.473191 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:15.473166    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47c095a-1345-4bb2-a1db-b718dcf7261f-kube-api-access-lwc9x" pod "a47c095a-1345-4bb2-a1db-b718dcf7261f" (UID: "a47c095a-1345-4bb2-a1db-b718dcf7261f"). InnerVolumeSpecName "kube-api-access-lwc9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:15.572555 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:15.572524    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47c095a-1345-4bb2-a1db-b718dcf7261f-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:15.572555 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:15.572552    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwc9x\" (UniqueName: \"kubernetes.io/projected/a47c095a-1345-4bb2-a1db-b718dcf7261f-kube-api-access-lwc9x\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:15.681147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:15.681067    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg"]
Nov 03 01:16:15.681293 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:15.681263    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg" podUID="c14b1d5d-e75b-45da-a948-9b0a9b7f855e" containerName="maas-api" containerID="cri-o://1b3c1a46dea803e1b6f37488504e75e6442a1e3647785206d8da6910523a5231" gracePeriod=30
Nov 03 01:16:15.977595 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:15.977567    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg"
Nov 03 01:16:16.077113 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.077081    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-kube-api-access-2gkv4\" (UniqueName: \"kubernetes.io/projected/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-kube-api-access-2gkv4\") pod \"c14b1d5d-e75b-45da-a948-9b0a9b7f855e\" (UID: \"c14b1d5d-e75b-45da-a948-9b0a9b7f855e\") "
Nov 03 01:16:16.077319 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.077168    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-openshift-service-ca\") pod \"c14b1d5d-e75b-45da-a948-9b0a9b7f855e\" (UID: \"c14b1d5d-e75b-45da-a948-9b0a9b7f855e\") "
Nov 03 01:16:16.077319 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.077212    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-maas-api-tls\") pod \"c14b1d5d-e75b-45da-a948-9b0a9b7f855e\" (UID: \"c14b1d5d-e75b-45da-a948-9b0a9b7f855e\") "
Nov 03 01:16:16.077319 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.077236    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-metrics-certs\") pod \"c14b1d5d-e75b-45da-a948-9b0a9b7f855e\" (UID: \"c14b1d5d-e75b-45da-a948-9b0a9b7f855e\") "
Nov 03 01:16:16.077582 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.077545    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-openshift-service-ca" pod "c14b1d5d-e75b-45da-a948-9b0a9b7f855e" (UID: "c14b1d5d-e75b-45da-a948-9b0a9b7f855e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:16:16.079258 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.079232    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-metrics-certs" pod "c14b1d5d-e75b-45da-a948-9b0a9b7f855e" (UID: "c14b1d5d-e75b-45da-a948-9b0a9b7f855e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:16:16.079375 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.079280    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-maas-api-tls" pod "c14b1d5d-e75b-45da-a948-9b0a9b7f855e" (UID: "c14b1d5d-e75b-45da-a948-9b0a9b7f855e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:16:16.079375 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.079334    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-kube-api-access-2gkv4" pod "c14b1d5d-e75b-45da-a948-9b0a9b7f855e" (UID: "c14b1d5d-e75b-45da-a948-9b0a9b7f855e"). InnerVolumeSpecName "kube-api-access-2gkv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:16.178660 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.178568    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:16.178660 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.178604    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:16.178660 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.178618    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gkv4\" (UniqueName: \"kubernetes.io/projected/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-kube-api-access-2gkv4\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:16.178660 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.178632    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c14b1d5d-e75b-45da-a948-9b0a9b7f855e-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:16.284450 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.284370    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2"
Nov 03 01:16:16.284450 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.284391    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2" event={"ID":"a47c095a-1345-4bb2-a1db-b718dcf7261f","Type":"ContainerDied","Data":"9730878c43971764aa3a3cc44417952c064125ffc8736cc7e0f0ef46cd794f2a"}
Nov 03 01:16:16.284676 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.284459    5012 scope.go:122] "RemoveContainer" containerID="00f2ad9e5dd2396b090eb3259b66a189c64310f268b23da2d70b0f6549dd4e02"
Nov 03 01:16:16.286009 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.285986    5012 generic.go:356] "Generic (PLEG): container finished" podID="c14b1d5d-e75b-45da-a948-9b0a9b7f855e" containerID="1b3c1a46dea803e1b6f37488504e75e6442a1e3647785206d8da6910523a5231" exitCode=0
Nov 03 01:16:16.286146 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.286130    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg" event={"ID":"c14b1d5d-e75b-45da-a948-9b0a9b7f855e","Type":"ContainerDied","Data":"1b3c1a46dea803e1b6f37488504e75e6442a1e3647785206d8da6910523a5231"}
Nov 03 01:16:16.286212 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.286175    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg" event={"ID":"c14b1d5d-e75b-45da-a948-9b0a9b7f855e","Type":"ContainerDied","Data":"0ae645c02a49f2741505c7c897ed5619696cdd8124d944c7159734fc17f73fe3"}
Nov 03 01:16:16.286278 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.286253    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg"
Nov 03 01:16:16.296206 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.296187    5012 scope.go:122] "RemoveContainer" containerID="1b3c1a46dea803e1b6f37488504e75e6442a1e3647785206d8da6910523a5231"
Nov 03 01:16:16.305573 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.305553    5012 scope.go:122] "RemoveContainer" containerID="1b3c1a46dea803e1b6f37488504e75e6442a1e3647785206d8da6910523a5231"
Nov 03 01:16:16.305838 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:16:16.305817    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3c1a46dea803e1b6f37488504e75e6442a1e3647785206d8da6910523a5231\": container with ID starting with 1b3c1a46dea803e1b6f37488504e75e6442a1e3647785206d8da6910523a5231 not found: ID does not exist" containerID="1b3c1a46dea803e1b6f37488504e75e6442a1e3647785206d8da6910523a5231"
Nov 03 01:16:16.305896 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.305846    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3c1a46dea803e1b6f37488504e75e6442a1e3647785206d8da6910523a5231"} err="failed to get container status \"1b3c1a46dea803e1b6f37488504e75e6442a1e3647785206d8da6910523a5231\": rpc error: code = NotFound desc = could not find container \"1b3c1a46dea803e1b6f37488504e75e6442a1e3647785206d8da6910523a5231\": container with ID starting with 1b3c1a46dea803e1b6f37488504e75e6442a1e3647785206d8da6910523a5231 not found: ID does not exist"
Nov 03 01:16:16.330726 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.330704    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2"]
Nov 03 01:16:16.342621 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.342593    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0db47e-979769587-ckzz2"]
Nov 03 01:16:16.365084 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.365049    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg"]
Nov 03 01:16:16.368038 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:16.368016    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0db47e-64597c8f9f-mlpjg"]
Nov 03 01:16:17.245747 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:17.245715    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a47c095a-1345-4bb2-a1db-b718dcf7261f" path="/var/lib/kubelet/pods/a47c095a-1345-4bb2-a1db-b718dcf7261f/volumes"
Nov 03 01:16:17.246212 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:17.246196    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c14b1d5d-e75b-45da-a948-9b0a9b7f855e" path="/var/lib/kubelet/pods/c14b1d5d-e75b-45da-a948-9b0a9b7f855e/volumes"
Nov 03 01:16:26.536151 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:26.536085    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt"]
Nov 03 01:16:26.536963 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:26.536370    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt" podUID="98a31bfd-3a54-4fac-b1ff-d768f6ec5504" containerName="maas-api" containerID="cri-o://a5e5ae4d3c0edc04fdc4d3ee216517b5c8a7896ec27765289394db5d777ddb1f" gracePeriod=30
Nov 03 01:16:26.850482 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:26.850448    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt"
Nov 03 01:16:26.919926 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:26.919881    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-kube-api-access-rcq4v\" (UniqueName: \"kubernetes.io/projected/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-kube-api-access-rcq4v\") pod \"98a31bfd-3a54-4fac-b1ff-d768f6ec5504\" (UID: \"98a31bfd-3a54-4fac-b1ff-d768f6ec5504\") "
Nov 03 01:16:26.920101 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:26.919987    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-metrics-certs\" (UniqueName: \"kubernetes.io/secret/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-metrics-certs\") pod \"98a31bfd-3a54-4fac-b1ff-d768f6ec5504\" (UID: \"98a31bfd-3a54-4fac-b1ff-d768f6ec5504\") "
Nov 03 01:16:26.920101 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:26.920026    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-maas-api-tls\") pod \"98a31bfd-3a54-4fac-b1ff-d768f6ec5504\" (UID: \"98a31bfd-3a54-4fac-b1ff-d768f6ec5504\") "
Nov 03 01:16:26.920236 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:26.920148    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-openshift-service-ca\") pod \"98a31bfd-3a54-4fac-b1ff-d768f6ec5504\" (UID: \"98a31bfd-3a54-4fac-b1ff-d768f6ec5504\") "
Nov 03 01:16:26.920577 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:26.920540    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-openshift-service-ca" pod "98a31bfd-3a54-4fac-b1ff-d768f6ec5504" (UID: "98a31bfd-3a54-4fac-b1ff-d768f6ec5504"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:16:26.922116 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:26.922088    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-metrics-certs" pod "98a31bfd-3a54-4fac-b1ff-d768f6ec5504" (UID: "98a31bfd-3a54-4fac-b1ff-d768f6ec5504"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:16:26.922247 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:26.922226    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-kube-api-access-rcq4v" pod "98a31bfd-3a54-4fac-b1ff-d768f6ec5504" (UID: "98a31bfd-3a54-4fac-b1ff-d768f6ec5504"). InnerVolumeSpecName "kube-api-access-rcq4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:26.922516 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:26.922500    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-maas-api-tls" pod "98a31bfd-3a54-4fac-b1ff-d768f6ec5504" (UID: "98a31bfd-3a54-4fac-b1ff-d768f6ec5504"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:16:27.027768 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:27.027634    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcq4v\" (UniqueName: \"kubernetes.io/projected/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-kube-api-access-rcq4v\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:27.027768 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:27.027691    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:27.027768 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:27.027714    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:27.027768 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:27.027732    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a31bfd-3a54-4fac-b1ff-d768f6ec5504-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:27.363144 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:27.363103    5012 generic.go:356] "Generic (PLEG): container finished" podID="98a31bfd-3a54-4fac-b1ff-d768f6ec5504" containerID="a5e5ae4d3c0edc04fdc4d3ee216517b5c8a7896ec27765289394db5d777ddb1f" exitCode=0
Nov 03 01:16:27.363356 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:27.363162    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt"
Nov 03 01:16:27.363356 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:27.363189    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt" event={"ID":"98a31bfd-3a54-4fac-b1ff-d768f6ec5504","Type":"ContainerDied","Data":"a5e5ae4d3c0edc04fdc4d3ee216517b5c8a7896ec27765289394db5d777ddb1f"}
Nov 03 01:16:27.363356 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:27.363251    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt" event={"ID":"98a31bfd-3a54-4fac-b1ff-d768f6ec5504","Type":"ContainerDied","Data":"84e24714688092a9da9179de2241fe3fe28cbab2879212f48bb0d6c50b6978bc"}
Nov 03 01:16:27.363356 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:27.363282    5012 scope.go:122] "RemoveContainer" containerID="a5e5ae4d3c0edc04fdc4d3ee216517b5c8a7896ec27765289394db5d777ddb1f"
Nov 03 01:16:27.377029 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:27.377009    5012 scope.go:122] "RemoveContainer" containerID="a5e5ae4d3c0edc04fdc4d3ee216517b5c8a7896ec27765289394db5d777ddb1f"
Nov 03 01:16:27.377319 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:16:27.377298    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5e5ae4d3c0edc04fdc4d3ee216517b5c8a7896ec27765289394db5d777ddb1f\": container with ID starting with a5e5ae4d3c0edc04fdc4d3ee216517b5c8a7896ec27765289394db5d777ddb1f not found: ID does not exist" containerID="a5e5ae4d3c0edc04fdc4d3ee216517b5c8a7896ec27765289394db5d777ddb1f"
Nov 03 01:16:27.377393 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:27.377333    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5e5ae4d3c0edc04fdc4d3ee216517b5c8a7896ec27765289394db5d777ddb1f"} err="failed to get container status \"a5e5ae4d3c0edc04fdc4d3ee216517b5c8a7896ec27765289394db5d777ddb1f\": rpc error: code = NotFound desc = could not find container \"a5e5ae4d3c0edc04fdc4d3ee216517b5c8a7896ec27765289394db5d777ddb1f\": container with ID starting with a5e5ae4d3c0edc04fdc4d3ee216517b5c8a7896ec27765289394db5d777ddb1f not found: ID does not exist"
Nov 03 01:16:27.408949 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:27.408921    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt"]
Nov 03 01:16:27.419445 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:27.419422    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9908e8-84df665985-tw2mt"]
Nov 03 01:16:29.243226 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:29.243151    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98a31bfd-3a54-4fac-b1ff-d768f6ec5504" path="/var/lib/kubelet/pods/98a31bfd-3a54-4fac-b1ff-d768f6ec5504/volumes"
Nov 03 01:16:30.772371 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:30.772327    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575"]
Nov 03 01:16:35.773332 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:35.773282    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575" podUID="7e3bfaef-0df4-4b55-9eba-3761cf145de0" containerName="payload-processing" containerID="cri-o://dfe1afd709b554e9063ffa60322d4c39380f691e82d80e6ee472e1d86563ce68" gracePeriod=25
Nov 03 01:16:37.363238 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:16:37.363203    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:16:39.286563 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:39.286528    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv"]
Nov 03 01:16:39.287137 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:39.286734    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv" podUID="a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22" containerName="maas-api" containerID="cri-o://c2f23afb85a9925d108df5879d073b731900d756e87326421707c5989498a21c" gracePeriod=30
Nov 03 01:16:39.449509 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:39.449472    5012 generic.go:356] "Generic (PLEG): container finished" podID="a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22" containerID="c2f23afb85a9925d108df5879d073b731900d756e87326421707c5989498a21c" exitCode=0
Nov 03 01:16:39.449672 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:39.449504    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv" event={"ID":"a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22","Type":"ContainerDied","Data":"c2f23afb85a9925d108df5879d073b731900d756e87326421707c5989498a21c"}
Nov 03 01:16:39.567741 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:39.567621    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv"
Nov 03 01:16:39.676504 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:39.676473    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-kube-api-access-5kwhd\" (UniqueName: \"kubernetes.io/projected/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-kube-api-access-5kwhd\") pod \"a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22\" (UID: \"a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22\") "
Nov 03 01:16:39.676690 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:39.676602    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-maas-api-tls\") pod \"a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22\" (UID: \"a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22\") "
Nov 03 01:16:39.676690 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:39.676670    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-metrics-certs\") pod \"a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22\" (UID: \"a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22\") "
Nov 03 01:16:39.676827 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:39.676737    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-openshift-service-ca\") pod \"a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22\" (UID: \"a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22\") "
Nov 03 01:16:39.677177 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:39.677150    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-openshift-service-ca" pod "a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22" (UID: "a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:16:39.677332 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:39.677268    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:39.678718 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:39.678685    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-maas-api-tls" pod "a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22" (UID: "a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:16:39.678842 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:39.678724    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-metrics-certs" pod "a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22" (UID: "a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:16:39.678842 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:39.678729    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-kube-api-access-5kwhd" pod "a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22" (UID: "a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22"). InnerVolumeSpecName "kube-api-access-5kwhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:39.778939 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:39.778883    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:39.779099 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:39.778947    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kwhd\" (UniqueName: \"kubernetes.io/projected/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-kube-api-access-5kwhd\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:39.779099 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:39.778959    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:39.964056 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:39.963999    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575" podUID="7e3bfaef-0df4-4b55-9eba-3761cf145de0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:16:40.456647 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:40.456606    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv"
Nov 03 01:16:40.457154 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:40.456646    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv" event={"ID":"a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22","Type":"ContainerDied","Data":"f24711105ab1e26452d6969ee4830505d1c3a318c43fd8ce444893bf3988a78c"}
Nov 03 01:16:40.457154 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:40.456718    5012 scope.go:122] "RemoveContainer" containerID="c2f23afb85a9925d108df5879d073b731900d756e87326421707c5989498a21c"
Nov 03 01:16:40.506791 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:40.506761    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv"]
Nov 03 01:16:40.514393 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:40.514370    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-099583-775dfd6fd7-bkbhv"]
Nov 03 01:16:41.244196 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:41.244161    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22" path="/var/lib/kubelet/pods/a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22/volumes"
Nov 03 01:16:44.960517 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:44.960477    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"]
Nov 03 01:16:44.961114 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:44.960726    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq" podUID="76d45dee-a04f-4d36-9496-6e6eef77b94a" containerName="istio-proxy" containerID="cri-o://d8dc42bd9980ad190890f74b528c6d9fe721e26dfd77ba7d38cc7215a6b241ac" gracePeriod=30
Nov 03 01:16:47.389379 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:47.389337    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"]
Nov 03 01:16:47.389839 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:47.389591    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd" podUID="cfe5fc6e-4689-49d4-8aa0-861317b1965f" containerName="istio-proxy" containerID="cri-o://c21a21bb579679b595176c35b4748a85b9058f734621ae3dd50c5d719bb30778" gracePeriod=30
Nov 03 01:16:49.951414 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:49.951377    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575" podUID="7e3bfaef-0df4-4b55-9eba-3761cf145de0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:16:50.231078 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.231049    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 01:16:50.384451 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.384421    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-workload-certs\") pod \"76d45dee-a04f-4d36-9496-6e6eef77b94a\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") "
Nov 03 01:16:50.384629 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.384482    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-token\" (UniqueName: \"kubernetes.io/projected/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-token\") pod \"76d45dee-a04f-4d36-9496-6e6eef77b94a\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") "
Nov 03 01:16:50.384629 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.384534    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-credential-socket\") pod \"76d45dee-a04f-4d36-9496-6e6eef77b94a\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") "
Nov 03 01:16:50.384629 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.384583    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-podinfo\") pod \"76d45dee-a04f-4d36-9496-6e6eef77b94a\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") "
Nov 03 01:16:50.384801 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.384649    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76d45dee-a04f-4d36-9496-6e6eef77b94a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d45dee-a04f-4d36-9496-6e6eef77b94a-istiod-ca-cert\") pod \"76d45dee-a04f-4d36-9496-6e6eef77b94a\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") "
Nov 03 01:16:50.384801 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.384679    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-envoy\") pod \"76d45dee-a04f-4d36-9496-6e6eef77b94a\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") "
Nov 03 01:16:50.384801 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.384726    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-workload-certs" pod "76d45dee-a04f-4d36-9496-6e6eef77b94a" (UID: "76d45dee-a04f-4d36-9496-6e6eef77b94a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:50.384801 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.384743    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-workload-socket\") pod \"76d45dee-a04f-4d36-9496-6e6eef77b94a\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") "
Nov 03 01:16:50.385066 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.384806    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-data\") pod \"76d45dee-a04f-4d36-9496-6e6eef77b94a\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") "
Nov 03 01:16:50.385066 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.384855    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76d45dee-a04f-4d36-9496-6e6eef77b94a-kube-api-access-s8b85\" (UniqueName: \"kubernetes.io/projected/76d45dee-a04f-4d36-9496-6e6eef77b94a-kube-api-access-s8b85\") pod \"76d45dee-a04f-4d36-9496-6e6eef77b94a\" (UID: \"76d45dee-a04f-4d36-9496-6e6eef77b94a\") "
Nov 03 01:16:50.385066 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.385012    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-credential-socket" pod "76d45dee-a04f-4d36-9496-6e6eef77b94a" (UID: "76d45dee-a04f-4d36-9496-6e6eef77b94a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:50.385066 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.385025    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-workload-socket" pod "76d45dee-a04f-4d36-9496-6e6eef77b94a" (UID: "76d45dee-a04f-4d36-9496-6e6eef77b94a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:50.385297 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.385116    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76d45dee-a04f-4d36-9496-6e6eef77b94a-istiod-ca-cert" pod "76d45dee-a04f-4d36-9496-6e6eef77b94a" (UID: "76d45dee-a04f-4d36-9496-6e6eef77b94a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:16:50.385297 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.385205    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-data" pod "76d45dee-a04f-4d36-9496-6e6eef77b94a" (UID: "76d45dee-a04f-4d36-9496-6e6eef77b94a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:50.385640 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.385617    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:50.385766 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.385722    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:50.385766 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.385744    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:50.385766 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.385759    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d45dee-a04f-4d36-9496-6e6eef77b94a-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:50.385995 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.385774    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:50.386999 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.386972    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-token" pod "76d45dee-a04f-4d36-9496-6e6eef77b94a" (UID: "76d45dee-a04f-4d36-9496-6e6eef77b94a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:50.387144 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.387099    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-envoy" pod "76d45dee-a04f-4d36-9496-6e6eef77b94a" (UID: "76d45dee-a04f-4d36-9496-6e6eef77b94a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:50.387226 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.387210    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-podinfo" pod "76d45dee-a04f-4d36-9496-6e6eef77b94a" (UID: "76d45dee-a04f-4d36-9496-6e6eef77b94a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:16:50.387321 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.387302    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d45dee-a04f-4d36-9496-6e6eef77b94a-kube-api-access-s8b85" pod "76d45dee-a04f-4d36-9496-6e6eef77b94a" (UID: "76d45dee-a04f-4d36-9496-6e6eef77b94a"). InnerVolumeSpecName "kube-api-access-s8b85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:50.486776 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.486717    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:50.486776 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.486742    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8b85\" (UniqueName: \"kubernetes.io/projected/76d45dee-a04f-4d36-9496-6e6eef77b94a-kube-api-access-s8b85\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:50.486776 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.486752    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:50.486776 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.486760    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d45dee-a04f-4d36-9496-6e6eef77b94a-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:50.521644 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.521616    5012 generic.go:356] "Generic (PLEG): container finished" podID="76d45dee-a04f-4d36-9496-6e6eef77b94a" containerID="d8dc42bd9980ad190890f74b528c6d9fe721e26dfd77ba7d38cc7215a6b241ac" exitCode=0
Nov 03 01:16:50.521744 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.521676    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"
Nov 03 01:16:50.521744 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.521694    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq" event={"ID":"76d45dee-a04f-4d36-9496-6e6eef77b94a","Type":"ContainerDied","Data":"d8dc42bd9980ad190890f74b528c6d9fe721e26dfd77ba7d38cc7215a6b241ac"}
Nov 03 01:16:50.521827 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.521747    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq" event={"ID":"76d45dee-a04f-4d36-9496-6e6eef77b94a","Type":"ContainerDied","Data":"e4e149c98a29b7825d3635e1d387f01805c48fb2b5e4c000a6dd7e145e6c413d"}
Nov 03 01:16:50.521827 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.521769    5012 scope.go:122] "RemoveContainer" containerID="d8dc42bd9980ad190890f74b528c6d9fe721e26dfd77ba7d38cc7215a6b241ac"
Nov 03 01:16:50.532724 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.532671    5012 scope.go:122] "RemoveContainer" containerID="b643215a0a0279fff23cb45def4f49d25a7d531f12a10a1bf8f698788b783a32"
Nov 03 01:16:50.541785 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.541767    5012 scope.go:122] "RemoveContainer" containerID="d8dc42bd9980ad190890f74b528c6d9fe721e26dfd77ba7d38cc7215a6b241ac"
Nov 03 01:16:50.542115 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:16:50.542097    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8dc42bd9980ad190890f74b528c6d9fe721e26dfd77ba7d38cc7215a6b241ac\": container with ID starting with d8dc42bd9980ad190890f74b528c6d9fe721e26dfd77ba7d38cc7215a6b241ac not found: ID does not exist" containerID="d8dc42bd9980ad190890f74b528c6d9fe721e26dfd77ba7d38cc7215a6b241ac"
Nov 03 01:16:50.542238 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.542121    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8dc42bd9980ad190890f74b528c6d9fe721e26dfd77ba7d38cc7215a6b241ac"} err="failed to get container status \"d8dc42bd9980ad190890f74b528c6d9fe721e26dfd77ba7d38cc7215a6b241ac\": rpc error: code = NotFound desc = could not find container \"d8dc42bd9980ad190890f74b528c6d9fe721e26dfd77ba7d38cc7215a6b241ac\": container with ID starting with d8dc42bd9980ad190890f74b528c6d9fe721e26dfd77ba7d38cc7215a6b241ac not found: ID does not exist"
Nov 03 01:16:50.542238 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.542136    5012 scope.go:122] "RemoveContainer" containerID="b643215a0a0279fff23cb45def4f49d25a7d531f12a10a1bf8f698788b783a32"
Nov 03 01:16:50.542396 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:16:50.542375    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b643215a0a0279fff23cb45def4f49d25a7d531f12a10a1bf8f698788b783a32\": container with ID starting with b643215a0a0279fff23cb45def4f49d25a7d531f12a10a1bf8f698788b783a32 not found: ID does not exist" containerID="b643215a0a0279fff23cb45def4f49d25a7d531f12a10a1bf8f698788b783a32"
Nov 03 01:16:50.542433 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.542402    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b643215a0a0279fff23cb45def4f49d25a7d531f12a10a1bf8f698788b783a32"} err="failed to get container status \"b643215a0a0279fff23cb45def4f49d25a7d531f12a10a1bf8f698788b783a32\": rpc error: code = NotFound desc = could not find container \"b643215a0a0279fff23cb45def4f49d25a7d531f12a10a1bf8f698788b783a32\": container with ID starting with b643215a0a0279fff23cb45def4f49d25a7d531f12a10a1bf8f698788b783a32 not found: ID does not exist"
Nov 03 01:16:50.569415 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.569388    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"]
Nov 03 01:16:50.579797 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.579774    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-7n5dq"]
Nov 03 01:16:50.849246 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.849214    5012 patch_prober.go:29] interesting pod/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:16:50.849434 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:50.849272    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd" podUID="cfe5fc6e-4689-49d4-8aa0-861317b1965f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:16:51.243299 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:51.243220    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76d45dee-a04f-4d36-9496-6e6eef77b94a" path="/var/lib/kubelet/pods/76d45dee-a04f-4d36-9496-6e6eef77b94a/volumes"
Nov 03 01:16:52.538499 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.538450    5012 generic.go:356] "Generic (PLEG): container finished" podID="cfe5fc6e-4689-49d4-8aa0-861317b1965f" containerID="c21a21bb579679b595176c35b4748a85b9058f734621ae3dd50c5d719bb30778" exitCode=0
Nov 03 01:16:52.538987 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.538528    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd" event={"ID":"cfe5fc6e-4689-49d4-8aa0-861317b1965f","Type":"ContainerDied","Data":"c21a21bb579679b595176c35b4748a85b9058f734621ae3dd50c5d719bb30778"}
Nov 03 01:16:52.538987 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.538613    5012 scope.go:122] "RemoveContainer" containerID="11a3b0c7e452e6df6e38322dbba7f28232631ff37acf03cf42122850dccf77b8"
Nov 03 01:16:52.657869 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.657842    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 01:16:52.724373 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.723877    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-podinfo\") pod \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") "
Nov 03 01:16:52.724724 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.724694    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-credential-socket\") pod \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") "
Nov 03 01:16:52.724820 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.724755    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-workload-certs\") pod \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") "
Nov 03 01:16:52.724820 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.724789    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istiod-ca-cert\") pod \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") "
Nov 03 01:16:52.724971 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.724846    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-token\" (UniqueName: \"kubernetes.io/projected/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-token\") pod \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") "
Nov 03 01:16:52.724971 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.724890    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-data\") pod \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") "
Nov 03 01:16:52.724971 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.724950    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfe5fc6e-4689-49d4-8aa0-861317b1965f-kube-api-access-22tdl\" (UniqueName: \"kubernetes.io/projected/cfe5fc6e-4689-49d4-8aa0-861317b1965f-kube-api-access-22tdl\") pod \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") "
Nov 03 01:16:52.725191 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.724977    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-workload-socket\") pod \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") "
Nov 03 01:16:52.725577 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.725554    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-data" pod "cfe5fc6e-4689-49d4-8aa0-861317b1965f" (UID: "cfe5fc6e-4689-49d4-8aa0-861317b1965f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:52.725695 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.725649    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-workload-certs" pod "cfe5fc6e-4689-49d4-8aa0-861317b1965f" (UID: "cfe5fc6e-4689-49d4-8aa0-861317b1965f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:52.726267 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.726038    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-credential-socket" pod "cfe5fc6e-4689-49d4-8aa0-861317b1965f" (UID: "cfe5fc6e-4689-49d4-8aa0-861317b1965f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:52.726267 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.726230    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istiod-ca-cert" pod "cfe5fc6e-4689-49d4-8aa0-861317b1965f" (UID: "cfe5fc6e-4689-49d4-8aa0-861317b1965f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:16:52.726517 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.726481    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-workload-socket" pod "cfe5fc6e-4689-49d4-8aa0-861317b1965f" (UID: "cfe5fc6e-4689-49d4-8aa0-861317b1965f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:52.728946 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.727964    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-podinfo" pod "cfe5fc6e-4689-49d4-8aa0-861317b1965f" (UID: "cfe5fc6e-4689-49d4-8aa0-861317b1965f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:16:52.732817 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.732775    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-token" pod "cfe5fc6e-4689-49d4-8aa0-861317b1965f" (UID: "cfe5fc6e-4689-49d4-8aa0-861317b1965f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:52.737106 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.737069    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe5fc6e-4689-49d4-8aa0-861317b1965f-kube-api-access-22tdl" pod "cfe5fc6e-4689-49d4-8aa0-861317b1965f" (UID: "cfe5fc6e-4689-49d4-8aa0-861317b1965f"). InnerVolumeSpecName "kube-api-access-22tdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:52.825838 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.825809    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-envoy\") pod \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\" (UID: \"cfe5fc6e-4689-49d4-8aa0-861317b1965f\") "
Nov 03 01:16:52.826086 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.826071    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:52.826138 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.826089    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:52.826138 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.826098    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:52.826138 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.826106    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22tdl\" (UniqueName: \"kubernetes.io/projected/cfe5fc6e-4689-49d4-8aa0-861317b1965f-kube-api-access-22tdl\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:52.826138 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.826114    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:52.826138 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.826123    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:52.826138 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.826132    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:52.826338 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.826140    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:52.827731 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.827709    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-envoy" pod "cfe5fc6e-4689-49d4-8aa0-861317b1965f" (UID: "cfe5fc6e-4689-49d4-8aa0-861317b1965f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:52.926703 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:52.926670    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe5fc6e-4689-49d4-8aa0-861317b1965f-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:53.546266 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:53.546184    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"
Nov 03 01:16:53.546266 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:53.546235    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd" event={"ID":"cfe5fc6e-4689-49d4-8aa0-861317b1965f","Type":"ContainerDied","Data":"80aad6f3e207312b5140cbe64c0ac46cd6321729f229a319b470a00ddfc3ae00"}
Nov 03 01:16:53.546774 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:53.546303    5012 scope.go:122] "RemoveContainer" containerID="c21a21bb579679b595176c35b4748a85b9058f734621ae3dd50c5d719bb30778"
Nov 03 01:16:53.603411 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:53.603382    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"]
Nov 03 01:16:53.613672 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:53.613646    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-69pcd"]
Nov 03 01:16:54.048753 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:54.048720    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb"]
Nov 03 01:16:54.048978 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:54.048939    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb" podUID="7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e" containerName="maas-api" containerID="cri-o://724cbab7237ae96b722e68e586db350e17b43ebc2d7cb003928ad264c04618eb" gracePeriod=30
Nov 03 01:16:54.339863 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:54.339835    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb"
Nov 03 01:16:54.437702 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:54.437671    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-maas-api-tls\") pod \"7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e\" (UID: \"7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e\") "
Nov 03 01:16:54.437865 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:54.437724    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-kube-api-access-kbbcc\" (UniqueName: \"kubernetes.io/projected/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-kube-api-access-kbbcc\") pod \"7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e\" (UID: \"7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e\") "
Nov 03 01:16:54.437865 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:54.437788    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-metrics-certs\") pod \"7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e\" (UID: \"7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e\") "
Nov 03 01:16:54.437865 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:54.437816    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-openshift-service-ca\") pod \"7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e\" (UID: \"7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e\") "
Nov 03 01:16:54.438266 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:54.438236    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-openshift-service-ca" pod "7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e" (UID: "7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:16:54.439859 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:54.439838    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-maas-api-tls" pod "7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e" (UID: "7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:16:54.439982 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:54.439893    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-kube-api-access-kbbcc" pod "7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e" (UID: "7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e"). InnerVolumeSpecName "kube-api-access-kbbcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:54.439982 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:54.439946    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-metrics-certs" pod "7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e" (UID: "7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:16:54.539386 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:54.539346    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:54.539386 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:54.539382    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbbcc\" (UniqueName: \"kubernetes.io/projected/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-kube-api-access-kbbcc\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:54.539386 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:54.539392    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:54.539751 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:54.539402    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:54.554061 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:54.554025    5012 generic.go:356] "Generic (PLEG): container finished" podID="7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e" containerID="724cbab7237ae96b722e68e586db350e17b43ebc2d7cb003928ad264c04618eb" exitCode=0
Nov 03 01:16:54.554558 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:54.554079    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb"
Nov 03 01:16:54.554558 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:54.554104    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb" event={"ID":"7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e","Type":"ContainerDied","Data":"724cbab7237ae96b722e68e586db350e17b43ebc2d7cb003928ad264c04618eb"}
Nov 03 01:16:54.554558 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:54.554165    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb" event={"ID":"7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e","Type":"ContainerDied","Data":"9818293a83a2c0bdf53dff24fb965ada9ce248eec51cf5e88b126176fd5191cf"}
Nov 03 01:16:54.554558 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:54.554192    5012 scope.go:122] "RemoveContainer" containerID="724cbab7237ae96b722e68e586db350e17b43ebc2d7cb003928ad264c04618eb"
Nov 03 01:16:54.564900 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:54.564882    5012 scope.go:122] "RemoveContainer" containerID="724cbab7237ae96b722e68e586db350e17b43ebc2d7cb003928ad264c04618eb"
Nov 03 01:16:54.565173 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:16:54.565154    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"724cbab7237ae96b722e68e586db350e17b43ebc2d7cb003928ad264c04618eb\": container with ID starting with 724cbab7237ae96b722e68e586db350e17b43ebc2d7cb003928ad264c04618eb not found: ID does not exist" containerID="724cbab7237ae96b722e68e586db350e17b43ebc2d7cb003928ad264c04618eb"
Nov 03 01:16:54.565243 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:54.565179    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"724cbab7237ae96b722e68e586db350e17b43ebc2d7cb003928ad264c04618eb"} err="failed to get container status \"724cbab7237ae96b722e68e586db350e17b43ebc2d7cb003928ad264c04618eb\": rpc error: code = NotFound desc = could not find container \"724cbab7237ae96b722e68e586db350e17b43ebc2d7cb003928ad264c04618eb\": container with ID starting with 724cbab7237ae96b722e68e586db350e17b43ebc2d7cb003928ad264c04618eb not found: ID does not exist"
Nov 03 01:16:54.602131 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:54.602095    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb"]
Nov 03 01:16:54.604768 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:54.604746    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5c109b-84c96b576-8plsb"]
Nov 03 01:16:55.243696 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:55.243653    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e" path="/var/lib/kubelet/pods/7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e/volumes"
Nov 03 01:16:55.244237 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:55.244213    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfe5fc6e-4689-49d4-8aa0-861317b1965f" path="/var/lib/kubelet/pods/cfe5fc6e-4689-49d4-8aa0-861317b1965f/volumes"
Nov 03 01:16:56.028243 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:56.028215    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575"
Nov 03 01:16:56.052243 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:56.052174    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e3bfaef-0df4-4b55-9eba-3761cf145de0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e3bfaef-0df4-4b55-9eba-3761cf145de0-plugins-config-volume\") pod \"7e3bfaef-0df4-4b55-9eba-3761cf145de0\" (UID: \"7e3bfaef-0df4-4b55-9eba-3761cf145de0\") "
Nov 03 01:16:56.052368 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:56.052273    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e3bfaef-0df4-4b55-9eba-3761cf145de0-kube-api-access-zbflx\" (UniqueName: \"kubernetes.io/projected/7e3bfaef-0df4-4b55-9eba-3761cf145de0-kube-api-access-zbflx\") pod \"7e3bfaef-0df4-4b55-9eba-3761cf145de0\" (UID: \"7e3bfaef-0df4-4b55-9eba-3761cf145de0\") "
Nov 03 01:16:56.052513 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:56.052490    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e3bfaef-0df4-4b55-9eba-3761cf145de0-plugins-config-volume" pod "7e3bfaef-0df4-4b55-9eba-3761cf145de0" (UID: "7e3bfaef-0df4-4b55-9eba-3761cf145de0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:16:56.052647 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:56.052627    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e3bfaef-0df4-4b55-9eba-3761cf145de0-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:56.054288 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:56.054267    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3bfaef-0df4-4b55-9eba-3761cf145de0-kube-api-access-zbflx" pod "7e3bfaef-0df4-4b55-9eba-3761cf145de0" (UID: "7e3bfaef-0df4-4b55-9eba-3761cf145de0"). InnerVolumeSpecName "kube-api-access-zbflx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:56.153365 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:56.153334    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbflx\" (UniqueName: \"kubernetes.io/projected/7e3bfaef-0df4-4b55-9eba-3761cf145de0-kube-api-access-zbflx\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:16:56.569189 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:56.569152    5012 generic.go:356] "Generic (PLEG): container finished" podID="7e3bfaef-0df4-4b55-9eba-3761cf145de0" containerID="dfe1afd709b554e9063ffa60322d4c39380f691e82d80e6ee472e1d86563ce68" exitCode=0
Nov 03 01:16:56.569371 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:56.569197    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575"
Nov 03 01:16:56.569371 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:56.569228    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575" event={"ID":"7e3bfaef-0df4-4b55-9eba-3761cf145de0","Type":"ContainerDied","Data":"dfe1afd709b554e9063ffa60322d4c39380f691e82d80e6ee472e1d86563ce68"}
Nov 03 01:16:56.569371 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:56.569285    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575" event={"ID":"7e3bfaef-0df4-4b55-9eba-3761cf145de0","Type":"ContainerDied","Data":"64170a62fc576a66913cbf9bc977132375a08bcc669a58dd536a1f9bb5073fd9"}
Nov 03 01:16:56.569371 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:56.569314    5012 scope.go:122] "RemoveContainer" containerID="dfe1afd709b554e9063ffa60322d4c39380f691e82d80e6ee472e1d86563ce68"
Nov 03 01:16:56.579800 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:56.579781    5012 scope.go:122] "RemoveContainer" containerID="dfe1afd709b554e9063ffa60322d4c39380f691e82d80e6ee472e1d86563ce68"
Nov 03 01:16:56.580111 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:16:56.580084    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe1afd709b554e9063ffa60322d4c39380f691e82d80e6ee472e1d86563ce68\": container with ID starting with dfe1afd709b554e9063ffa60322d4c39380f691e82d80e6ee472e1d86563ce68 not found: ID does not exist" containerID="dfe1afd709b554e9063ffa60322d4c39380f691e82d80e6ee472e1d86563ce68"
Nov 03 01:16:56.580209 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:56.580113    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe1afd709b554e9063ffa60322d4c39380f691e82d80e6ee472e1d86563ce68"} err="failed to get container status \"dfe1afd709b554e9063ffa60322d4c39380f691e82d80e6ee472e1d86563ce68\": rpc error: code = NotFound desc = could not find container \"dfe1afd709b554e9063ffa60322d4c39380f691e82d80e6ee472e1d86563ce68\": container with ID starting with dfe1afd709b554e9063ffa60322d4c39380f691e82d80e6ee472e1d86563ce68 not found: ID does not exist"
Nov 03 01:16:56.619373 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:56.619344    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575"]
Nov 03 01:16:56.631135 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:56.631109    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-099583-b94d76f8c-r9575"]
Nov 03 01:16:57.243364 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:57.243330    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e3bfaef-0df4-4b55-9eba-3761cf145de0" path="/var/lib/kubelet/pods/7e3bfaef-0df4-4b55-9eba-3761cf145de0/volumes"
Nov 03 01:16:58.117801 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.117758    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"]
Nov 03 01:16:58.118033 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.118004    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs" podUID="b260d86b-87b8-4779-b2cb-02c6aa18b9f1" containerName="istio-proxy" containerID="cri-o://cb5d68de573cf4e4c8d968c10ce2b548c0c1b9bc68950f0e4ebfd49255773b01" gracePeriod=30
Nov 03 01:16:58.996434 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.996395    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c02b89e9-4kwrs"]
Nov 03 01:16:58.997079 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997057    5012 cpu_manager.go:414] "containerMap: removing container" podUID="a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22" containerName="maas-api"
Nov 03 01:16:58.997180 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997081    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22" containerName="maas-api"
Nov 03 01:16:58.997180 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997097    5012 cpu_manager.go:414] "containerMap: removing container" podUID="26130d50-27b4-4de9-8ec5-03af2ae23cbc" containerName="maas-api"
Nov 03 01:16:58.997180 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997105    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26130d50-27b4-4de9-8ec5-03af2ae23cbc" containerName="maas-api"
Nov 03 01:16:58.997180 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997124    5012 cpu_manager.go:414] "containerMap: removing container" podUID="7e3bfaef-0df4-4b55-9eba-3761cf145de0" containerName="payload-processing"
Nov 03 01:16:58.997180 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997132    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e3bfaef-0df4-4b55-9eba-3761cf145de0" containerName="payload-processing"
Nov 03 01:16:58.997180 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997149    5012 cpu_manager.go:414] "containerMap: removing container" podUID="7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e" containerName="maas-api"
Nov 03 01:16:58.997180 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997156    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e" containerName="maas-api"
Nov 03 01:16:58.997180 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997175    5012 cpu_manager.go:414] "containerMap: removing container" podUID="a47c095a-1345-4bb2-a1db-b718dcf7261f" containerName="payload-pre-processing"
Nov 03 01:16:58.997604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997183    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47c095a-1345-4bb2-a1db-b718dcf7261f" containerName="payload-pre-processing"
Nov 03 01:16:58.997604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997197    5012 cpu_manager.go:414] "containerMap: removing container" podUID="c3c993ca-dfa2-422f-bb9b-21af3056b405" containerName="cleanup"
Nov 03 01:16:58.997604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997204    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c993ca-dfa2-422f-bb9b-21af3056b405" containerName="cleanup"
Nov 03 01:16:58.997604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997212    5012 cpu_manager.go:414] "containerMap: removing container" podUID="c14b1d5d-e75b-45da-a948-9b0a9b7f855e" containerName="maas-api"
Nov 03 01:16:58.997604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997218    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c14b1d5d-e75b-45da-a948-9b0a9b7f855e" containerName="maas-api"
Nov 03 01:16:58.997604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997228    5012 cpu_manager.go:414] "containerMap: removing container" podUID="76d45dee-a04f-4d36-9496-6e6eef77b94a" containerName="istio-proxy"
Nov 03 01:16:58.997604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997237    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76d45dee-a04f-4d36-9496-6e6eef77b94a" containerName="istio-proxy"
Nov 03 01:16:58.997604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997253    5012 cpu_manager.go:414] "containerMap: removing container" podUID="98a31bfd-3a54-4fac-b1ff-d768f6ec5504" containerName="maas-api"
Nov 03 01:16:58.997604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997260    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98a31bfd-3a54-4fac-b1ff-d768f6ec5504" containerName="maas-api"
Nov 03 01:16:58.997604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997271    5012 cpu_manager.go:414] "containerMap: removing container" podUID="cfe5fc6e-4689-49d4-8aa0-861317b1965f" containerName="istio-proxy"
Nov 03 01:16:58.997604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997277    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfe5fc6e-4689-49d4-8aa0-861317b1965f" containerName="istio-proxy"
Nov 03 01:16:58.997604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997303    5012 cpu_manager.go:414] "containerMap: removing container" podUID="cfe5fc6e-4689-49d4-8aa0-861317b1965f" containerName="istio-proxy"
Nov 03 01:16:58.997604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997343    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfe5fc6e-4689-49d4-8aa0-861317b1965f" containerName="istio-proxy"
Nov 03 01:16:58.997604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997367    5012 cpu_manager.go:414] "containerMap: removing container" podUID="76d45dee-a04f-4d36-9496-6e6eef77b94a" containerName="istio-proxy"
Nov 03 01:16:58.997604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997440    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76d45dee-a04f-4d36-9496-6e6eef77b94a" containerName="istio-proxy"
Nov 03 01:16:58.997604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997463    5012 cpu_manager.go:414] "containerMap: removing container" podUID="0d206e64-ac15-4522-846a-e6c10863f235" containerName="istio-proxy"
Nov 03 01:16:58.997604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997472    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d206e64-ac15-4522-846a-e6c10863f235" containerName="istio-proxy"
Nov 03 01:16:58.997604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997597    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="c14b1d5d-e75b-45da-a948-9b0a9b7f855e" containerName="maas-api"
Nov 03 01:16:58.997604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997612    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="76d45dee-a04f-4d36-9496-6e6eef77b94a" containerName="istio-proxy"
Nov 03 01:16:58.998579 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997622    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="98a31bfd-3a54-4fac-b1ff-d768f6ec5504" containerName="maas-api"
Nov 03 01:16:58.998579 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997631    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfe5fc6e-4689-49d4-8aa0-861317b1965f" containerName="istio-proxy"
Nov 03 01:16:58.998579 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997643    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfe5fc6e-4689-49d4-8aa0-861317b1965f" containerName="istio-proxy"
Nov 03 01:16:58.998579 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997656    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="76d45dee-a04f-4d36-9496-6e6eef77b94a" containerName="istio-proxy"
Nov 03 01:16:58.998579 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997666    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d206e64-ac15-4522-846a-e6c10863f235" containerName="istio-proxy"
Nov 03 01:16:58.998579 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997674    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6d013a8-d1ca-4ff2-8bb9-bbe5fba4de22" containerName="maas-api"
Nov 03 01:16:58.998579 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997683    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="26130d50-27b4-4de9-8ec5-03af2ae23cbc" containerName="maas-api"
Nov 03 01:16:58.998579 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997694    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e3bfaef-0df4-4b55-9eba-3761cf145de0" containerName="payload-processing"
Nov 03 01:16:58.998579 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997703    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a4f41bb-b2f7-46d2-95e5-a265a55bdb0e" containerName="maas-api"
Nov 03 01:16:58.998579 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997714    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c26a270-d736-4678-9ff8-bfabcb06fe44" containerName="cleanup"
Nov 03 01:16:58.998579 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:58.997723    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47c095a-1345-4bb2-a1db-b718dcf7261f" containerName="payload-pre-processing"
Nov 03 01:16:59.002703 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:59.002680    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02b89e9-4kwrs"
Nov 03 01:16:59.019829 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:59.019801    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c02b89e9-4kwrs"]
Nov 03 01:16:59.026687 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:59.026657    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:16:59.026813 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:59.026691    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:16:59.026813 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:59.026776    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:16:59.028030 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:59.028011    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:16:59.106419 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:59.106392    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69e14a92-1722-49b6-b26a-f45d08375476-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c02b89e9-4kwrs\" (UID: \"69e14a92-1722-49b6-b26a-f45d08375476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02b89e9-4kwrs"
Nov 03 01:16:59.106542 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:59.106452    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctr8j\" (UniqueName: \"kubernetes.io/projected/69e14a92-1722-49b6-b26a-f45d08375476-kube-api-access-ctr8j\") pod \"maas-api-revoke-sub-c02b89e9-4kwrs\" (UID: \"69e14a92-1722-49b6-b26a-f45d08375476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02b89e9-4kwrs"
Nov 03 01:16:59.106610 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:59.106586    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69e14a92-1722-49b6-b26a-f45d08375476-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c02b89e9-4kwrs\" (UID: \"69e14a92-1722-49b6-b26a-f45d08375476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02b89e9-4kwrs"
Nov 03 01:16:59.208056 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:59.208019    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctr8j\" (UniqueName: \"kubernetes.io/projected/69e14a92-1722-49b6-b26a-f45d08375476-kube-api-access-ctr8j\") pod \"maas-api-revoke-sub-c02b89e9-4kwrs\" (UID: \"69e14a92-1722-49b6-b26a-f45d08375476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02b89e9-4kwrs"
Nov 03 01:16:59.208262 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:59.208104    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69e14a92-1722-49b6-b26a-f45d08375476-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c02b89e9-4kwrs\" (UID: \"69e14a92-1722-49b6-b26a-f45d08375476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02b89e9-4kwrs"
Nov 03 01:16:59.208262 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:59.208157    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69e14a92-1722-49b6-b26a-f45d08375476-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c02b89e9-4kwrs\" (UID: \"69e14a92-1722-49b6-b26a-f45d08375476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02b89e9-4kwrs"
Nov 03 01:16:59.208833 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:59.208804    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69e14a92-1722-49b6-b26a-f45d08375476-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c02b89e9-4kwrs\" (UID: \"69e14a92-1722-49b6-b26a-f45d08375476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02b89e9-4kwrs"
Nov 03 01:16:59.210585 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:59.210565    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69e14a92-1722-49b6-b26a-f45d08375476-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c02b89e9-4kwrs\" (UID: \"69e14a92-1722-49b6-b26a-f45d08375476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02b89e9-4kwrs"
Nov 03 01:16:59.227415 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:59.227390    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctr8j\" (UniqueName: \"kubernetes.io/projected/69e14a92-1722-49b6-b26a-f45d08375476-kube-api-access-ctr8j\") pod \"maas-api-revoke-sub-c02b89e9-4kwrs\" (UID: \"69e14a92-1722-49b6-b26a-f45d08375476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02b89e9-4kwrs"
Nov 03 01:16:59.316948 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:59.316870    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02b89e9-4kwrs"
Nov 03 01:16:59.619037 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:59.619009    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c02b89e9-4kwrs"]
Nov 03 01:16:59.619935 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:16:59.619900    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e14a92_1722_49b6_b26a_f45d08375476.slice/crio-0f8a1be6386f041a2bd8f4408369a8e9f52128f5b640a1cb81c84b9270d45f37 WatchSource:0}: Error finding container 0f8a1be6386f041a2bd8f4408369a8e9f52128f5b640a1cb81c84b9270d45f37: Status 404 returned error can't find the container with id 0f8a1be6386f041a2bd8f4408369a8e9f52128f5b640a1cb81c84b9270d45f37
Nov 03 01:16:59.800200 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:59.800170    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78"]
Nov 03 01:16:59.842526 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:16:59.842491    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd"]
Nov 03 01:17:00.599103 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:00.599073    5012 generic.go:356] "Generic (PLEG): container finished" podID="69e14a92-1722-49b6-b26a-f45d08375476" containerID="072cc46a2b589c884ffa1f369ee6f78951cf6b0bc575bfa36bbd1fdff10f34b1" exitCode=0
Nov 03 01:17:00.599452 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:00.599111    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02b89e9-4kwrs" event={"ID":"69e14a92-1722-49b6-b26a-f45d08375476","Type":"ContainerDied","Data":"072cc46a2b589c884ffa1f369ee6f78951cf6b0bc575bfa36bbd1fdff10f34b1"}
Nov 03 01:17:00.599452 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:00.599166    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02b89e9-4kwrs" event={"ID":"69e14a92-1722-49b6-b26a-f45d08375476","Type":"ContainerStarted","Data":"0f8a1be6386f041a2bd8f4408369a8e9f52128f5b640a1cb81c84b9270d45f37"}
Nov 03 01:17:01.750000 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:01.749971    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02b89e9-4kwrs"
Nov 03 01:17:01.839141 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:01.839111    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69e14a92-1722-49b6-b26a-f45d08375476-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69e14a92-1722-49b6-b26a-f45d08375476-maas-api-openshift-ca\") pod \"69e14a92-1722-49b6-b26a-f45d08375476\" (UID: \"69e14a92-1722-49b6-b26a-f45d08375476\") "
Nov 03 01:17:01.839316 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:01.839216    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69e14a92-1722-49b6-b26a-f45d08375476-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69e14a92-1722-49b6-b26a-f45d08375476-maas-api-xks-ca\") pod \"69e14a92-1722-49b6-b26a-f45d08375476\" (UID: \"69e14a92-1722-49b6-b26a-f45d08375476\") "
Nov 03 01:17:01.839316 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:01.839242    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69e14a92-1722-49b6-b26a-f45d08375476-kube-api-access-ctr8j\" (UniqueName: \"kubernetes.io/projected/69e14a92-1722-49b6-b26a-f45d08375476-kube-api-access-ctr8j\") pod \"69e14a92-1722-49b6-b26a-f45d08375476\" (UID: \"69e14a92-1722-49b6-b26a-f45d08375476\") "
Nov 03 01:17:01.839524 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:01.839499    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e14a92-1722-49b6-b26a-f45d08375476-maas-api-openshift-ca" pod "69e14a92-1722-49b6-b26a-f45d08375476" (UID: "69e14a92-1722-49b6-b26a-f45d08375476"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:17:01.841387 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:01.841363    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e14a92-1722-49b6-b26a-f45d08375476-kube-api-access-ctr8j" pod "69e14a92-1722-49b6-b26a-f45d08375476" (UID: "69e14a92-1722-49b6-b26a-f45d08375476"). InnerVolumeSpecName "kube-api-access-ctr8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:01.841583 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:01.841555    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69e14a92-1722-49b6-b26a-f45d08375476-maas-api-xks-ca" pod "69e14a92-1722-49b6-b26a-f45d08375476" (UID: "69e14a92-1722-49b6-b26a-f45d08375476"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:17:01.940493 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:01.940433    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctr8j\" (UniqueName: \"kubernetes.io/projected/69e14a92-1722-49b6-b26a-f45d08375476-kube-api-access-ctr8j\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:01.940493 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:01.940460    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69e14a92-1722-49b6-b26a-f45d08375476-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:01.940493 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:01.940471    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69e14a92-1722-49b6-b26a-f45d08375476-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:02.614976 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:02.614939    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02b89e9-4kwrs"
Nov 03 01:17:02.614976 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:02.614955    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02b89e9-4kwrs" event={"ID":"69e14a92-1722-49b6-b26a-f45d08375476","Type":"ContainerDied","Data":"0f8a1be6386f041a2bd8f4408369a8e9f52128f5b640a1cb81c84b9270d45f37"}
Nov 03 01:17:02.615205 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:02.615007    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f8a1be6386f041a2bd8f4408369a8e9f52128f5b640a1cb81c84b9270d45f37"
Nov 03 01:17:03.382276 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.382253    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:17:03.454497 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.454423    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-data\") pod \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") "
Nov 03 01:17:03.454497 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.454474    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-workload-certs\") pod \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") "
Nov 03 01:17:03.454699 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.454503    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-workload-socket\") pod \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") "
Nov 03 01:17:03.454699 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.454523    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-credential-socket\") pod \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") "
Nov 03 01:17:03.454699 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.454559    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istiod-ca-cert\") pod \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") "
Nov 03 01:17:03.454699 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.454603    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-envoy\") pod \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") "
Nov 03 01:17:03.454947 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.454773    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-podinfo\") pod \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") "
Nov 03 01:17:03.454947 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.454781    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-data" pod "b260d86b-87b8-4779-b2cb-02c6aa18b9f1" (UID: "b260d86b-87b8-4779-b2cb-02c6aa18b9f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:03.454947 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.454809    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-token\" (UniqueName: \"kubernetes.io/projected/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-token\") pod \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") "
Nov 03 01:17:03.454947 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.454840    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-kube-api-access-l65zk\" (UniqueName: \"kubernetes.io/projected/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-kube-api-access-l65zk\") pod \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\" (UID: \"b260d86b-87b8-4779-b2cb-02c6aa18b9f1\") "
Nov 03 01:17:03.455178 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.454945    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-workload-certs" pod "b260d86b-87b8-4779-b2cb-02c6aa18b9f1" (UID: "b260d86b-87b8-4779-b2cb-02c6aa18b9f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:03.455178 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.454961    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-credential-socket" pod "b260d86b-87b8-4779-b2cb-02c6aa18b9f1" (UID: "b260d86b-87b8-4779-b2cb-02c6aa18b9f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:03.455299 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.455188    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-workload-socket" pod "b260d86b-87b8-4779-b2cb-02c6aa18b9f1" (UID: "b260d86b-87b8-4779-b2cb-02c6aa18b9f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:03.455357 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.455324    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:03.455357 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.455346    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:03.455464 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.455361    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:03.455464 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.455364    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istiod-ca-cert" pod "b260d86b-87b8-4779-b2cb-02c6aa18b9f1" (UID: "b260d86b-87b8-4779-b2cb-02c6aa18b9f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:17:03.457107 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.457079    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-envoy" pod "b260d86b-87b8-4779-b2cb-02c6aa18b9f1" (UID: "b260d86b-87b8-4779-b2cb-02c6aa18b9f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:03.457221 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.457124    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-token" pod "b260d86b-87b8-4779-b2cb-02c6aa18b9f1" (UID: "b260d86b-87b8-4779-b2cb-02c6aa18b9f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:03.457297 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.457284    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-kube-api-access-l65zk" pod "b260d86b-87b8-4779-b2cb-02c6aa18b9f1" (UID: "b260d86b-87b8-4779-b2cb-02c6aa18b9f1"). InnerVolumeSpecName "kube-api-access-l65zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:03.457575 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.457549    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-podinfo" pod "b260d86b-87b8-4779-b2cb-02c6aa18b9f1" (UID: "b260d86b-87b8-4779-b2cb-02c6aa18b9f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:17:03.555867 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.555834    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:03.555867 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.555862    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:03.555867 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.555871    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l65zk\" (UniqueName: \"kubernetes.io/projected/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-kube-api-access-l65zk\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:03.556100 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.555882    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:03.556100 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.555891    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:03.556100 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.555899    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b260d86b-87b8-4779-b2cb-02c6aa18b9f1-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:03.622662 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.622626    5012 generic.go:356] "Generic (PLEG): container finished" podID="b260d86b-87b8-4779-b2cb-02c6aa18b9f1" containerID="cb5d68de573cf4e4c8d968c10ce2b548c0c1b9bc68950f0e4ebfd49255773b01" exitCode=0
Nov 03 01:17:03.622793 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.622679    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"
Nov 03 01:17:03.622793 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.622714    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs" event={"ID":"b260d86b-87b8-4779-b2cb-02c6aa18b9f1","Type":"ContainerDied","Data":"cb5d68de573cf4e4c8d968c10ce2b548c0c1b9bc68950f0e4ebfd49255773b01"}
Nov 03 01:17:03.622793 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.622773    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs" event={"ID":"b260d86b-87b8-4779-b2cb-02c6aa18b9f1","Type":"ContainerDied","Data":"6f1f629b97bb8b3c7443d8d1e97e9236a7afd5ad0fd462a9774f131bad79e8bb"}
Nov 03 01:17:03.622958 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.622794    5012 scope.go:122] "RemoveContainer" containerID="cb5d68de573cf4e4c8d968c10ce2b548c0c1b9bc68950f0e4ebfd49255773b01"
Nov 03 01:17:03.633451 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.633435    5012 scope.go:122] "RemoveContainer" containerID="cb5d68de573cf4e4c8d968c10ce2b548c0c1b9bc68950f0e4ebfd49255773b01"
Nov 03 01:17:03.633699 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:17:03.633680    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb5d68de573cf4e4c8d968c10ce2b548c0c1b9bc68950f0e4ebfd49255773b01\": container with ID starting with cb5d68de573cf4e4c8d968c10ce2b548c0c1b9bc68950f0e4ebfd49255773b01 not found: ID does not exist" containerID="cb5d68de573cf4e4c8d968c10ce2b548c0c1b9bc68950f0e4ebfd49255773b01"
Nov 03 01:17:03.633771 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.633706    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb5d68de573cf4e4c8d968c10ce2b548c0c1b9bc68950f0e4ebfd49255773b01"} err="failed to get container status \"cb5d68de573cf4e4c8d968c10ce2b548c0c1b9bc68950f0e4ebfd49255773b01\": rpc error: code = NotFound desc = could not find container \"cb5d68de573cf4e4c8d968c10ce2b548c0c1b9bc68950f0e4ebfd49255773b01\": container with ID starting with cb5d68de573cf4e4c8d968c10ce2b548c0c1b9bc68950f0e4ebfd49255773b01 not found: ID does not exist"
Nov 03 01:17:03.671946 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.671922    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"]
Nov 03 01:17:03.675478 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:03.675456    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vrgzs"]
Nov 03 01:17:04.553789 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:04.553738    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"]
Nov 03 01:17:04.801313 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:04.801263    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78" podUID="36bbffa3-d72d-4f83-8916-d2d70377d2ea" containerName="payload-processing" containerID="cri-o://eb84e8fe61df28aef04fe6c108b335306d0db57fe322d19cadbb146c0b7854ec" gracePeriod=25
Nov 03 01:17:04.843548 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:04.843466    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd" podUID="b420bc74-3aaf-4937-9237-9b35dc51ba8e" containerName="payload-pre-processing" containerID="cri-o://26a81f010246ba9b37556b94909b8c05c36f8c7a0d595de4ebbefb3008b7ca7c" gracePeriod=25
Nov 03 01:17:05.243425 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:05.243381    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b260d86b-87b8-4779-b2cb-02c6aa18b9f1" path="/var/lib/kubelet/pods/b260d86b-87b8-4779-b2cb-02c6aa18b9f1/volumes"
Nov 03 01:17:05.837341 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:05.837297    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd" podUID="b420bc74-3aaf-4937-9237-9b35dc51ba8e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:17:05.838262 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:05.838225    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78" podUID="36bbffa3-d72d-4f83-8916-d2d70377d2ea" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:17:07.368459 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:17:07.368414    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:17:10.395015 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:10.394977    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p"]
Nov 03 01:17:10.426931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:10.426877    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td"]
Nov 03 01:17:10.824669 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:10.824569    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b46b34e6-h2988"]
Nov 03 01:17:10.825338 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:10.825296    5012 cpu_manager.go:414] "containerMap: removing container" podUID="69e14a92-1722-49b6-b26a-f45d08375476" containerName="revoke-keys"
Nov 03 01:17:10.825554 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:10.825519    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69e14a92-1722-49b6-b26a-f45d08375476" containerName="revoke-keys"
Nov 03 01:17:10.825680 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:10.825582    5012 cpu_manager.go:414] "containerMap: removing container" podUID="b260d86b-87b8-4779-b2cb-02c6aa18b9f1" containerName="istio-proxy"
Nov 03 01:17:10.825680 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:10.825592    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b260d86b-87b8-4779-b2cb-02c6aa18b9f1" containerName="istio-proxy"
Nov 03 01:17:10.825788 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:10.825770    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="69e14a92-1722-49b6-b26a-f45d08375476" containerName="revoke-keys"
Nov 03 01:17:10.825840 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:10.825786    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="b260d86b-87b8-4779-b2cb-02c6aa18b9f1" containerName="istio-proxy"
Nov 03 01:17:10.831589 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:10.831566    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46b34e6-h2988"
Nov 03 01:17:10.846328 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:10.846298    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:17:10.847215 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:10.847182    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:17:10.847436 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:10.847420    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:17:10.847637 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:10.847615    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:17:10.854167 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:10.854135    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b46b34e6-h2988"]
Nov 03 01:17:10.934842 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:10.934813    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b46b34e6-h2988\" (UID: \"bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46b34e6-h2988"
Nov 03 01:17:10.934990 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:10.934966    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jlzh\" (UniqueName: \"kubernetes.io/projected/bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2-kube-api-access-6jlzh\") pod \"maas-api-revoke-sub-b46b34e6-h2988\" (UID: \"bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46b34e6-h2988"
Nov 03 01:17:10.935067 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:10.935050    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b46b34e6-h2988\" (UID: \"bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46b34e6-h2988"
Nov 03 01:17:11.036613 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:11.036566    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b46b34e6-h2988\" (UID: \"bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46b34e6-h2988"
Nov 03 01:17:11.036794 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:11.036661    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b46b34e6-h2988\" (UID: \"bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46b34e6-h2988"
Nov 03 01:17:11.036794 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:11.036783    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jlzh\" (UniqueName: \"kubernetes.io/projected/bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2-kube-api-access-6jlzh\") pod \"maas-api-revoke-sub-b46b34e6-h2988\" (UID: \"bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46b34e6-h2988"
Nov 03 01:17:11.037518 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:11.037489    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b46b34e6-h2988\" (UID: \"bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46b34e6-h2988"
Nov 03 01:17:11.039063 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:11.039041    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b46b34e6-h2988\" (UID: \"bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46b34e6-h2988"
Nov 03 01:17:11.053583 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:11.053561    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jlzh\" (UniqueName: \"kubernetes.io/projected/bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2-kube-api-access-6jlzh\") pod \"maas-api-revoke-sub-b46b34e6-h2988\" (UID: \"bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46b34e6-h2988"
Nov 03 01:17:11.167186 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:11.167161    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46b34e6-h2988"
Nov 03 01:17:11.461338 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:11.461305    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b46b34e6-h2988"]
Nov 03 01:17:11.462192 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:17:11.462167    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7b9c5a_6bd5_4921_b7f7_44fb44b4bec2.slice/crio-b5f25b622ed728d7ee5983014fcaafa33d02530d2b41db736afde2b52ede6a6b WatchSource:0}: Error finding container b5f25b622ed728d7ee5983014fcaafa33d02530d2b41db736afde2b52ede6a6b: Status 404 returned error can't find the container with id b5f25b622ed728d7ee5983014fcaafa33d02530d2b41db736afde2b52ede6a6b
Nov 03 01:17:11.681896 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:11.681860    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46b34e6-h2988" event={"ID":"bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2","Type":"ContainerStarted","Data":"b5f25b622ed728d7ee5983014fcaafa33d02530d2b41db736afde2b52ede6a6b"}
Nov 03 01:17:12.689628 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:12.689547    5012 generic.go:356] "Generic (PLEG): container finished" podID="bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2" containerID="eb446078f097a6d9058990610dfd04b86f6c8cfce50a51cf251d01160b86d2b7" exitCode=0
Nov 03 01:17:12.690004 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:12.689638    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46b34e6-h2988" event={"ID":"bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2","Type":"ContainerDied","Data":"eb446078f097a6d9058990610dfd04b86f6c8cfce50a51cf251d01160b86d2b7"}
Nov 03 01:17:13.839468 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:13.839441    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46b34e6-h2988"
Nov 03 01:17:13.976817 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:13.976741    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2-maas-api-openshift-ca\") pod \"bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2\" (UID: \"bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2\") "
Nov 03 01:17:13.976817 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:13.976795    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2-kube-api-access-6jlzh\" (UniqueName: \"kubernetes.io/projected/bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2-kube-api-access-6jlzh\") pod \"bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2\" (UID: \"bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2\") "
Nov 03 01:17:13.977057 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:13.976877    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2-maas-api-xks-ca\") pod \"bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2\" (UID: \"bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2\") "
Nov 03 01:17:13.977257 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:13.977229    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2-maas-api-openshift-ca" pod "bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2" (UID: "bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:17:13.978978 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:13.978953    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2-kube-api-access-6jlzh" pod "bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2" (UID: "bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2"). InnerVolumeSpecName "kube-api-access-6jlzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:13.979101 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:13.978998    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2-maas-api-xks-ca" pod "bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2" (UID: "bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:17:14.077635 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:14.077606    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:14.077635 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:14.077633    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:14.077802 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:14.077644    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jlzh\" (UniqueName: \"kubernetes.io/projected/bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2-kube-api-access-6jlzh\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:14.706232 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:14.706197    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46b34e6-h2988"
Nov 03 01:17:14.706232 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:14.706218    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46b34e6-h2988" event={"ID":"bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2","Type":"ContainerDied","Data":"b5f25b622ed728d7ee5983014fcaafa33d02530d2b41db736afde2b52ede6a6b"}
Nov 03 01:17:14.706447 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:14.706270    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5f25b622ed728d7ee5983014fcaafa33d02530d2b41db736afde2b52ede6a6b"
Nov 03 01:17:14.721433 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:17:14.721403    5012 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7b9c5a_6bd5_4921_b7f7_44fb44b4bec2.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 01:17:15.395646 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:15.395602    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p" podUID="d7409925-56ce-416e-b0dc-2a4fde810094" containerName="payload-processing" containerID="cri-o://226791cb1e23a28fca582c76c1c60dabbc3e10ed9d6cb7fb3d86635c0ff098b5" gracePeriod=25
Nov 03 01:17:15.428131 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:15.428090    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td" podUID="ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e" containerName="payload-pre-processing" containerID="cri-o://3e10ec59a42f642736aab46f85569ab4e5b3a34aedb8f5a3ea337831f3619786" gracePeriod=25
Nov 03 01:17:15.654125 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:15.654060    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p"
Nov 03 01:17:15.716924 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:15.716869    5012 generic.go:356] "Generic (PLEG): container finished" podID="d7409925-56ce-416e-b0dc-2a4fde810094" containerID="226791cb1e23a28fca582c76c1c60dabbc3e10ed9d6cb7fb3d86635c0ff098b5" exitCode=0
Nov 03 01:17:15.717076 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:15.716932    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p"
Nov 03 01:17:15.717076 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:15.716966    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p" event={"ID":"d7409925-56ce-416e-b0dc-2a4fde810094","Type":"ContainerDied","Data":"226791cb1e23a28fca582c76c1c60dabbc3e10ed9d6cb7fb3d86635c0ff098b5"}
Nov 03 01:17:15.717076 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:15.717035    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p" event={"ID":"d7409925-56ce-416e-b0dc-2a4fde810094","Type":"ContainerDied","Data":"3dcc34e41f182502d2ff48cb7136ac9000fac40896bd9bb5388128e6c53c9c96"}
Nov 03 01:17:15.717076 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:15.717068    5012 scope.go:122] "RemoveContainer" containerID="226791cb1e23a28fca582c76c1c60dabbc3e10ed9d6cb7fb3d86635c0ff098b5"
Nov 03 01:17:15.728190 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:15.728167    5012 scope.go:122] "RemoveContainer" containerID="226791cb1e23a28fca582c76c1c60dabbc3e10ed9d6cb7fb3d86635c0ff098b5"
Nov 03 01:17:15.728497 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:17:15.728474    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"226791cb1e23a28fca582c76c1c60dabbc3e10ed9d6cb7fb3d86635c0ff098b5\": container with ID starting with 226791cb1e23a28fca582c76c1c60dabbc3e10ed9d6cb7fb3d86635c0ff098b5 not found: ID does not exist" containerID="226791cb1e23a28fca582c76c1c60dabbc3e10ed9d6cb7fb3d86635c0ff098b5"
Nov 03 01:17:15.728576 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:15.728508    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"226791cb1e23a28fca582c76c1c60dabbc3e10ed9d6cb7fb3d86635c0ff098b5"} err="failed to get container status \"226791cb1e23a28fca582c76c1c60dabbc3e10ed9d6cb7fb3d86635c0ff098b5\": rpc error: code = NotFound desc = could not find container \"226791cb1e23a28fca582c76c1c60dabbc3e10ed9d6cb7fb3d86635c0ff098b5\": container with ID starting with 226791cb1e23a28fca582c76c1c60dabbc3e10ed9d6cb7fb3d86635c0ff098b5 not found: ID does not exist"
Nov 03 01:17:15.796020 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:15.795990    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7409925-56ce-416e-b0dc-2a4fde810094-kube-api-access-fm99m\" (UniqueName: \"kubernetes.io/projected/d7409925-56ce-416e-b0dc-2a4fde810094-kube-api-access-fm99m\") pod \"d7409925-56ce-416e-b0dc-2a4fde810094\" (UID: \"d7409925-56ce-416e-b0dc-2a4fde810094\") "
Nov 03 01:17:15.796212 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:15.796151    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7409925-56ce-416e-b0dc-2a4fde810094-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7409925-56ce-416e-b0dc-2a4fde810094-plugins-config-volume\") pod \"d7409925-56ce-416e-b0dc-2a4fde810094\" (UID: \"d7409925-56ce-416e-b0dc-2a4fde810094\") "
Nov 03 01:17:15.796497 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:15.796476    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7409925-56ce-416e-b0dc-2a4fde810094-plugins-config-volume" pod "d7409925-56ce-416e-b0dc-2a4fde810094" (UID: "d7409925-56ce-416e-b0dc-2a4fde810094"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:17:15.798001 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:15.797981    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7409925-56ce-416e-b0dc-2a4fde810094-kube-api-access-fm99m" pod "d7409925-56ce-416e-b0dc-2a4fde810094" (UID: "d7409925-56ce-416e-b0dc-2a4fde810094"). InnerVolumeSpecName "kube-api-access-fm99m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:15.837216 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:15.837174    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd" podUID="b420bc74-3aaf-4937-9237-9b35dc51ba8e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:17:15.837375 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:15.837212    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78" podUID="36bbffa3-d72d-4f83-8916-d2d70377d2ea" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:17:15.897712 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:15.897681    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm99m\" (UniqueName: \"kubernetes.io/projected/d7409925-56ce-416e-b0dc-2a4fde810094-kube-api-access-fm99m\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:15.897712 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:15.897707    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7409925-56ce-416e-b0dc-2a4fde810094-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:16.059543 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:16.059501    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p"]
Nov 03 01:17:16.063092 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:16.063059    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b85343-657658f49f-ddn8p"]
Nov 03 01:17:17.246682 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:17.246641    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7409925-56ce-416e-b0dc-2a4fde810094" path="/var/lib/kubelet/pods/d7409925-56ce-416e-b0dc-2a4fde810094/volumes"
Nov 03 01:17:19.575319 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:19.575281    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" containerID="cri-o://a175a22f87acfac139b9e246b43af2f1da0354dde5232c86e1395002d3fabe03" gracePeriod=45
Nov 03 01:17:19.740150 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:19.740104    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": dial tcp 10.132.0.87:8000: connect: connection refused"
Nov 03 01:17:19.754084 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:19.754048    5012 generic.go:356] "Generic (PLEG): container finished" podID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerID="a175a22f87acfac139b9e246b43af2f1da0354dde5232c86e1395002d3fabe03" exitCode=0
Nov 03 01:17:19.754208 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:19.754114    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" event={"ID":"ba41fa63-4d99-49a5-aa53-5b15efddb4a5","Type":"ContainerDied","Data":"a175a22f87acfac139b9e246b43af2f1da0354dde5232c86e1395002d3fabe03"}
Nov 03 01:17:19.821998 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:19.821972    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 01:17:19.843413 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:19.843360    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-home\" (UniqueName: \"kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-home\") pod \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\" (UID: \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\") "
Nov 03 01:17:19.843413 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:19.843394    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-dshm\") pod \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\" (UID: \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\") "
Nov 03 01:17:19.843573 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:19.843454    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-tmp-dir\") pod \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\" (UID: \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\") "
Nov 03 01:17:19.843573 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:19.843510    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-model-cache\") pod \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\" (UID: \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\") "
Nov 03 01:17:19.843695 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:19.843572    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-tls-certs\") pod \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\" (UID: \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\") "
Nov 03 01:17:19.843695 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:19.843599    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-kube-api-access-698zz\" (UniqueName: \"kubernetes.io/projected/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-kube-api-access-698zz\") pod \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\" (UID: \"ba41fa63-4d99-49a5-aa53-5b15efddb4a5\") "
Nov 03 01:17:19.843695 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:19.843595    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-home" pod "ba41fa63-4d99-49a5-aa53-5b15efddb4a5" (UID: "ba41fa63-4d99-49a5-aa53-5b15efddb4a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:19.843869 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:19.843721    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-tmp-dir" pod "ba41fa63-4d99-49a5-aa53-5b15efddb4a5" (UID: "ba41fa63-4d99-49a5-aa53-5b15efddb4a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:19.843869 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:19.843834    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-model-cache" pod "ba41fa63-4d99-49a5-aa53-5b15efddb4a5" (UID: "ba41fa63-4d99-49a5-aa53-5b15efddb4a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:19.844093 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:19.844074    5012 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-model-cache\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:19.844151 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:19.844097    5012 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-home\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:19.844151 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:19.844110    5012 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-tmp-dir\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:19.845688 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:19.845662    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-dshm" pod "ba41fa63-4d99-49a5-aa53-5b15efddb4a5" (UID: "ba41fa63-4d99-49a5-aa53-5b15efddb4a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:19.845797 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:19.845716    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-tls-certs" pod "ba41fa63-4d99-49a5-aa53-5b15efddb4a5" (UID: "ba41fa63-4d99-49a5-aa53-5b15efddb4a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:17:19.845797 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:19.845747    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-kube-api-access-698zz" pod "ba41fa63-4d99-49a5-aa53-5b15efddb4a5" (UID: "ba41fa63-4d99-49a5-aa53-5b15efddb4a5"). InnerVolumeSpecName "kube-api-access-698zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:19.945583 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:19.945552    5012 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-dshm\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:19.945583 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:19.945580    5012 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-tls-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:19.945785 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:19.945592    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-698zz\" (UniqueName: \"kubernetes.io/projected/ba41fa63-4d99-49a5-aa53-5b15efddb4a5-kube-api-access-698zz\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:20.762187 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:20.762153    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"
Nov 03 01:17:20.762715 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:20.762186    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv" event={"ID":"ba41fa63-4d99-49a5-aa53-5b15efddb4a5","Type":"ContainerDied","Data":"8d8bab76cd9c6f968d2114c2311660547bc70b913c2bd1ca46c7f1eb4ffe79a9"}
Nov 03 01:17:20.762715 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:20.762242    5012 scope.go:122] "RemoveContainer" containerID="a175a22f87acfac139b9e246b43af2f1da0354dde5232c86e1395002d3fabe03"
Nov 03 01:17:20.809867 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:20.809842    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"]
Nov 03 01:17:20.819761 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:20.819737    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b85343/premium-w1-b85343-kserve-5c5c7dcf4d-sdqbv"]
Nov 03 01:17:21.242481 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:21.242439    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" path="/var/lib/kubelet/pods/ba41fa63-4d99-49a5-aa53-5b15efddb4a5/volumes"
Nov 03 01:17:24.273015 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:24.272971    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td" podUID="ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:17:25.102616 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.102583    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78"
Nov 03 01:17:25.135965 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.135930    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd"
Nov 03 01:17:25.200148 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.199819    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b420bc74-3aaf-4937-9237-9b35dc51ba8e-kube-api-access-nj9h9\" (UniqueName: \"kubernetes.io/projected/b420bc74-3aaf-4937-9237-9b35dc51ba8e-kube-api-access-nj9h9\") pod \"b420bc74-3aaf-4937-9237-9b35dc51ba8e\" (UID: \"b420bc74-3aaf-4937-9237-9b35dc51ba8e\") "
Nov 03 01:17:25.200148 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.199953    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36bbffa3-d72d-4f83-8916-d2d70377d2ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36bbffa3-d72d-4f83-8916-d2d70377d2ea-plugins-config-volume\") pod \"36bbffa3-d72d-4f83-8916-d2d70377d2ea\" (UID: \"36bbffa3-d72d-4f83-8916-d2d70377d2ea\") "
Nov 03 01:17:25.200148 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.200049    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36bbffa3-d72d-4f83-8916-d2d70377d2ea-kube-api-access-b7p5k\" (UniqueName: \"kubernetes.io/projected/36bbffa3-d72d-4f83-8916-d2d70377d2ea-kube-api-access-b7p5k\") pod \"36bbffa3-d72d-4f83-8916-d2d70377d2ea\" (UID: \"36bbffa3-d72d-4f83-8916-d2d70377d2ea\") "
Nov 03 01:17:25.200496 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.200445    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36bbffa3-d72d-4f83-8916-d2d70377d2ea-plugins-config-volume" pod "36bbffa3-d72d-4f83-8916-d2d70377d2ea" (UID: "36bbffa3-d72d-4f83-8916-d2d70377d2ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:17:25.201014 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.200539    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b420bc74-3aaf-4937-9237-9b35dc51ba8e-plugins-config-volume" pod "b420bc74-3aaf-4937-9237-9b35dc51ba8e" (UID: "b420bc74-3aaf-4937-9237-9b35dc51ba8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:17:25.201014 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.200589    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b420bc74-3aaf-4937-9237-9b35dc51ba8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b420bc74-3aaf-4937-9237-9b35dc51ba8e-plugins-config-volume\") pod \"b420bc74-3aaf-4937-9237-9b35dc51ba8e\" (UID: \"b420bc74-3aaf-4937-9237-9b35dc51ba8e\") "
Nov 03 01:17:25.201014 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.200967    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b420bc74-3aaf-4937-9237-9b35dc51ba8e-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:25.201014 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.200990    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36bbffa3-d72d-4f83-8916-d2d70377d2ea-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:25.202423 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.202389    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b420bc74-3aaf-4937-9237-9b35dc51ba8e-kube-api-access-nj9h9" pod "b420bc74-3aaf-4937-9237-9b35dc51ba8e" (UID: "b420bc74-3aaf-4937-9237-9b35dc51ba8e"). InnerVolumeSpecName "kube-api-access-nj9h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:25.202526 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.202435    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36bbffa3-d72d-4f83-8916-d2d70377d2ea-kube-api-access-b7p5k" pod "36bbffa3-d72d-4f83-8916-d2d70377d2ea" (UID: "36bbffa3-d72d-4f83-8916-d2d70377d2ea"). InnerVolumeSpecName "kube-api-access-b7p5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:25.301975 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.301896    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj9h9\" (UniqueName: \"kubernetes.io/projected/b420bc74-3aaf-4937-9237-9b35dc51ba8e-kube-api-access-nj9h9\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:25.301975 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.301970    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7p5k\" (UniqueName: \"kubernetes.io/projected/36bbffa3-d72d-4f83-8916-d2d70377d2ea-kube-api-access-b7p5k\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:25.796809 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.796764    5012 generic.go:356] "Generic (PLEG): container finished" podID="b420bc74-3aaf-4937-9237-9b35dc51ba8e" containerID="26a81f010246ba9b37556b94909b8c05c36f8c7a0d595de4ebbefb3008b7ca7c" exitCode=0
Nov 03 01:17:25.797067 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.796818    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd"
Nov 03 01:17:25.797067 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.796851    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd" event={"ID":"b420bc74-3aaf-4937-9237-9b35dc51ba8e","Type":"ContainerDied","Data":"26a81f010246ba9b37556b94909b8c05c36f8c7a0d595de4ebbefb3008b7ca7c"}
Nov 03 01:17:25.797067 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.796939    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd" event={"ID":"b420bc74-3aaf-4937-9237-9b35dc51ba8e","Type":"ContainerDied","Data":"2e5b5e39e49f997508e82ef637fb13ff49eb731181fc78c51a4bbad28d75dd23"}
Nov 03 01:17:25.797067 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.796969    5012 scope.go:122] "RemoveContainer" containerID="26a81f010246ba9b37556b94909b8c05c36f8c7a0d595de4ebbefb3008b7ca7c"
Nov 03 01:17:25.799491 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.799464    5012 generic.go:356] "Generic (PLEG): container finished" podID="36bbffa3-d72d-4f83-8916-d2d70377d2ea" containerID="eb84e8fe61df28aef04fe6c108b335306d0db57fe322d19cadbb146c0b7854ec" exitCode=0
Nov 03 01:17:25.799628 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.799517    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78"
Nov 03 01:17:25.799628 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.799531    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78" event={"ID":"36bbffa3-d72d-4f83-8916-d2d70377d2ea","Type":"ContainerDied","Data":"eb84e8fe61df28aef04fe6c108b335306d0db57fe322d19cadbb146c0b7854ec"}
Nov 03 01:17:25.799628 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.799574    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78" event={"ID":"36bbffa3-d72d-4f83-8916-d2d70377d2ea","Type":"ContainerDied","Data":"29877e9ab5566b0564404cd95d103ea7e7381bb09e152653922a8f980ee191d3"}
Nov 03 01:17:25.808591 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.808561    5012 scope.go:122] "RemoveContainer" containerID="26a81f010246ba9b37556b94909b8c05c36f8c7a0d595de4ebbefb3008b7ca7c"
Nov 03 01:17:25.808948 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:17:25.808899    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26a81f010246ba9b37556b94909b8c05c36f8c7a0d595de4ebbefb3008b7ca7c\": container with ID starting with 26a81f010246ba9b37556b94909b8c05c36f8c7a0d595de4ebbefb3008b7ca7c not found: ID does not exist" containerID="26a81f010246ba9b37556b94909b8c05c36f8c7a0d595de4ebbefb3008b7ca7c"
Nov 03 01:17:25.809044 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.808956    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26a81f010246ba9b37556b94909b8c05c36f8c7a0d595de4ebbefb3008b7ca7c"} err="failed to get container status \"26a81f010246ba9b37556b94909b8c05c36f8c7a0d595de4ebbefb3008b7ca7c\": rpc error: code = NotFound desc = could not find container \"26a81f010246ba9b37556b94909b8c05c36f8c7a0d595de4ebbefb3008b7ca7c\": container with ID starting with 26a81f010246ba9b37556b94909b8c05c36f8c7a0d595de4ebbefb3008b7ca7c not found: ID does not exist"
Nov 03 01:17:25.809044 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.808976    5012 scope.go:122] "RemoveContainer" containerID="eb84e8fe61df28aef04fe6c108b335306d0db57fe322d19cadbb146c0b7854ec"
Nov 03 01:17:25.826850 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.826824    5012 scope.go:122] "RemoveContainer" containerID="eb84e8fe61df28aef04fe6c108b335306d0db57fe322d19cadbb146c0b7854ec"
Nov 03 01:17:25.827222 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:17:25.827199    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb84e8fe61df28aef04fe6c108b335306d0db57fe322d19cadbb146c0b7854ec\": container with ID starting with eb84e8fe61df28aef04fe6c108b335306d0db57fe322d19cadbb146c0b7854ec not found: ID does not exist" containerID="eb84e8fe61df28aef04fe6c108b335306d0db57fe322d19cadbb146c0b7854ec"
Nov 03 01:17:25.827319 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.827236    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb84e8fe61df28aef04fe6c108b335306d0db57fe322d19cadbb146c0b7854ec"} err="failed to get container status \"eb84e8fe61df28aef04fe6c108b335306d0db57fe322d19cadbb146c0b7854ec\": rpc error: code = NotFound desc = could not find container \"eb84e8fe61df28aef04fe6c108b335306d0db57fe322d19cadbb146c0b7854ec\": container with ID starting with eb84e8fe61df28aef04fe6c108b335306d0db57fe322d19cadbb146c0b7854ec not found: ID does not exist"
Nov 03 01:17:25.844014 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.843970    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd"]
Nov 03 01:17:25.854197 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.854160    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-99bdb8-7668c8f687-lk2wd"]
Nov 03 01:17:25.877198 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.877149    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78"]
Nov 03 01:17:25.881269 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:25.881236    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-99bdb8-84d64f99f5-kcp78"]
Nov 03 01:17:27.243132 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:27.243097    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36bbffa3-d72d-4f83-8916-d2d70377d2ea" path="/var/lib/kubelet/pods/36bbffa3-d72d-4f83-8916-d2d70377d2ea/volumes"
Nov 03 01:17:27.243514 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:27.243478    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b420bc74-3aaf-4937-9237-9b35dc51ba8e" path="/var/lib/kubelet/pods/b420bc74-3aaf-4937-9237-9b35dc51ba8e/volumes"
Nov 03 01:17:32.073717 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:32.073682    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"]
Nov 03 01:17:32.074236 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:32.073951    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd" podUID="f474b6f6-e47d-4c13-8f63-b188ca20fce6" containerName="istio-proxy" containerID="cri-o://aa2ab1d62981a8870cc3945d8098f07f188d7eb45cb8dc8d0a6b72aca8b05304" gracePeriod=30
Nov 03 01:17:34.273174 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:34.273136    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td" podUID="ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:17:35.687021 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:35.686954    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td"
Nov 03 01:17:35.833691 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:35.832659    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e-kube-api-access-gxfc4\" (UniqueName: \"kubernetes.io/projected/ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e-kube-api-access-gxfc4\") pod \"ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e\" (UID: \"ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e\") "
Nov 03 01:17:35.833691 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:35.833138    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e-plugins-config-volume\") pod \"ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e\" (UID: \"ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e\") "
Nov 03 01:17:35.834175 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:35.834121    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e-plugins-config-volume" pod "ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e" (UID: "ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:17:35.835461 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:35.835423    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e-kube-api-access-gxfc4" pod "ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e" (UID: "ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e"). InnerVolumeSpecName "kube-api-access-gxfc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:35.884493 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:35.880648    5012 generic.go:356] "Generic (PLEG): container finished" podID="ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e" containerID="3e10ec59a42f642736aab46f85569ab4e5b3a34aedb8f5a3ea337831f3619786" exitCode=0
Nov 03 01:17:35.884493 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:35.880711    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td" event={"ID":"ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e","Type":"ContainerDied","Data":"3e10ec59a42f642736aab46f85569ab4e5b3a34aedb8f5a3ea337831f3619786"}
Nov 03 01:17:35.884493 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:35.880760    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td" event={"ID":"ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e","Type":"ContainerDied","Data":"0c3737a318eda3495653db42fc2bc6e2c0506b4ba31525c3ee3eb5d49fa6c8ec"}
Nov 03 01:17:35.884493 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:35.880789    5012 scope.go:122] "RemoveContainer" containerID="3e10ec59a42f642736aab46f85569ab4e5b3a34aedb8f5a3ea337831f3619786"
Nov 03 01:17:35.884493 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:35.880967    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td"
Nov 03 01:17:35.922113 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:35.918201    5012 scope.go:122] "RemoveContainer" containerID="3e10ec59a42f642736aab46f85569ab4e5b3a34aedb8f5a3ea337831f3619786"
Nov 03 01:17:35.922113 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:17:35.918664    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e10ec59a42f642736aab46f85569ab4e5b3a34aedb8f5a3ea337831f3619786\": container with ID starting with 3e10ec59a42f642736aab46f85569ab4e5b3a34aedb8f5a3ea337831f3619786 not found: ID does not exist" containerID="3e10ec59a42f642736aab46f85569ab4e5b3a34aedb8f5a3ea337831f3619786"
Nov 03 01:17:35.922113 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:35.918726    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e10ec59a42f642736aab46f85569ab4e5b3a34aedb8f5a3ea337831f3619786"} err="failed to get container status \"3e10ec59a42f642736aab46f85569ab4e5b3a34aedb8f5a3ea337831f3619786\": rpc error: code = NotFound desc = could not find container \"3e10ec59a42f642736aab46f85569ab4e5b3a34aedb8f5a3ea337831f3619786\": container with ID starting with 3e10ec59a42f642736aab46f85569ab4e5b3a34aedb8f5a3ea337831f3619786 not found: ID does not exist"
Nov 03 01:17:35.936943 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:35.934514    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxfc4\" (UniqueName: \"kubernetes.io/projected/ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e-kube-api-access-gxfc4\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:35.936943 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:35.934552    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:35.960989 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:35.959519    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td"]
Nov 03 01:17:35.963928 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:35.963881    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b85343-69fbff6447-d54td"]
Nov 03 01:17:37.242499 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.242455    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e" path="/var/lib/kubelet/pods/ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e/volumes"
Nov 03 01:17:37.343177 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.343155    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 01:17:37.361926 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:17:37.361847    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:17:37.450033 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.449990    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-workload-certs\") pod \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") "
Nov 03 01:17:37.450225 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.450053    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-envoy\") pod \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") "
Nov 03 01:17:37.450225 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.450080    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-token\" (UniqueName: \"kubernetes.io/projected/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-token\") pod \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") "
Nov 03 01:17:37.450225 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.450108    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-workload-socket\") pod \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") "
Nov 03 01:17:37.450225 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.450208    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f474b6f6-e47d-4c13-8f63-b188ca20fce6-kube-api-access-mvdjh\" (UniqueName: \"kubernetes.io/projected/f474b6f6-e47d-4c13-8f63-b188ca20fce6-kube-api-access-mvdjh\") pod \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") "
Nov 03 01:17:37.450484 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.450252    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-data\") pod \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") "
Nov 03 01:17:37.450484 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.450313    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-podinfo\") pod \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") "
Nov 03 01:17:37.450484 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.450376    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-workload-certs" pod "f474b6f6-e47d-4c13-8f63-b188ca20fce6" (UID: "f474b6f6-e47d-4c13-8f63-b188ca20fce6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:37.450714 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.450685    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-credential-socket\") pod \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") "
Nov 03 01:17:37.450800 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.450704    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-data" pod "f474b6f6-e47d-4c13-8f63-b188ca20fce6" (UID: "f474b6f6-e47d-4c13-8f63-b188ca20fce6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:37.450800 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.450766    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istiod-ca-cert\") pod \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\" (UID: \"f474b6f6-e47d-4c13-8f63-b188ca20fce6\") "
Nov 03 01:17:37.450990 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.450960    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-credential-socket" pod "f474b6f6-e47d-4c13-8f63-b188ca20fce6" (UID: "f474b6f6-e47d-4c13-8f63-b188ca20fce6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:37.451106 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.451085    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-workload-socket" pod "f474b6f6-e47d-4c13-8f63-b188ca20fce6" (UID: "f474b6f6-e47d-4c13-8f63-b188ca20fce6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:37.451273 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.451246    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istiod-ca-cert" pod "f474b6f6-e47d-4c13-8f63-b188ca20fce6" (UID: "f474b6f6-e47d-4c13-8f63-b188ca20fce6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:17:37.451356 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.451329    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:37.451420 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.451353    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:37.451420 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.451369    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:37.451420 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.451383    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:37.452561 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.452534    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-podinfo" pod "f474b6f6-e47d-4c13-8f63-b188ca20fce6" (UID: "f474b6f6-e47d-4c13-8f63-b188ca20fce6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:17:37.452833 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.452808    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f474b6f6-e47d-4c13-8f63-b188ca20fce6-kube-api-access-mvdjh" pod "f474b6f6-e47d-4c13-8f63-b188ca20fce6" (UID: "f474b6f6-e47d-4c13-8f63-b188ca20fce6"). InnerVolumeSpecName "kube-api-access-mvdjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:37.452935 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.452851    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-envoy" pod "f474b6f6-e47d-4c13-8f63-b188ca20fce6" (UID: "f474b6f6-e47d-4c13-8f63-b188ca20fce6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:37.453054 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.453037    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-token" pod "f474b6f6-e47d-4c13-8f63-b188ca20fce6" (UID: "f474b6f6-e47d-4c13-8f63-b188ca20fce6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:37.552760 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.552714    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvdjh\" (UniqueName: \"kubernetes.io/projected/f474b6f6-e47d-4c13-8f63-b188ca20fce6-kube-api-access-mvdjh\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:37.552760 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.552755    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:37.552760 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.552765    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:37.553063 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.552774    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:37.553063 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.552783    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f474b6f6-e47d-4c13-8f63-b188ca20fce6-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:17:37.897334 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.897296    5012 generic.go:356] "Generic (PLEG): container finished" podID="f474b6f6-e47d-4c13-8f63-b188ca20fce6" containerID="aa2ab1d62981a8870cc3945d8098f07f188d7eb45cb8dc8d0a6b72aca8b05304" exitCode=0
Nov 03 01:17:37.897555 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.897347    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"
Nov 03 01:17:37.897555 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.897379    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd" event={"ID":"f474b6f6-e47d-4c13-8f63-b188ca20fce6","Type":"ContainerDied","Data":"aa2ab1d62981a8870cc3945d8098f07f188d7eb45cb8dc8d0a6b72aca8b05304"}
Nov 03 01:17:37.897555 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.897431    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd" event={"ID":"f474b6f6-e47d-4c13-8f63-b188ca20fce6","Type":"ContainerDied","Data":"5371106a9747a08fa18b3852a892b9effe08bd7b51913c638e6ddbc80bfee4df"}
Nov 03 01:17:37.897555 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.897461    5012 scope.go:122] "RemoveContainer" containerID="aa2ab1d62981a8870cc3945d8098f07f188d7eb45cb8dc8d0a6b72aca8b05304"
Nov 03 01:17:37.908076 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.908055    5012 scope.go:122] "RemoveContainer" containerID="72608dab623518b5be1416a046137a2d785eabfa85fd43ea68eda7942a2737c4"
Nov 03 01:17:37.917086 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.917065    5012 scope.go:122] "RemoveContainer" containerID="aa2ab1d62981a8870cc3945d8098f07f188d7eb45cb8dc8d0a6b72aca8b05304"
Nov 03 01:17:37.917356 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:17:37.917336    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa2ab1d62981a8870cc3945d8098f07f188d7eb45cb8dc8d0a6b72aca8b05304\": container with ID starting with aa2ab1d62981a8870cc3945d8098f07f188d7eb45cb8dc8d0a6b72aca8b05304 not found: ID does not exist" containerID="aa2ab1d62981a8870cc3945d8098f07f188d7eb45cb8dc8d0a6b72aca8b05304"
Nov 03 01:17:37.917420 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.917364    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa2ab1d62981a8870cc3945d8098f07f188d7eb45cb8dc8d0a6b72aca8b05304"} err="failed to get container status \"aa2ab1d62981a8870cc3945d8098f07f188d7eb45cb8dc8d0a6b72aca8b05304\": rpc error: code = NotFound desc = could not find container \"aa2ab1d62981a8870cc3945d8098f07f188d7eb45cb8dc8d0a6b72aca8b05304\": container with ID starting with aa2ab1d62981a8870cc3945d8098f07f188d7eb45cb8dc8d0a6b72aca8b05304 not found: ID does not exist"
Nov 03 01:17:37.917420 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.917382    5012 scope.go:122] "RemoveContainer" containerID="72608dab623518b5be1416a046137a2d785eabfa85fd43ea68eda7942a2737c4"
Nov 03 01:17:37.917646 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:17:37.917625    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72608dab623518b5be1416a046137a2d785eabfa85fd43ea68eda7942a2737c4\": container with ID starting with 72608dab623518b5be1416a046137a2d785eabfa85fd43ea68eda7942a2737c4 not found: ID does not exist" containerID="72608dab623518b5be1416a046137a2d785eabfa85fd43ea68eda7942a2737c4"
Nov 03 01:17:37.917701 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.917656    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72608dab623518b5be1416a046137a2d785eabfa85fd43ea68eda7942a2737c4"} err="failed to get container status \"72608dab623518b5be1416a046137a2d785eabfa85fd43ea68eda7942a2737c4\": rpc error: code = NotFound desc = could not find container \"72608dab623518b5be1416a046137a2d785eabfa85fd43ea68eda7942a2737c4\": container with ID starting with 72608dab623518b5be1416a046137a2d785eabfa85fd43ea68eda7942a2737c4 not found: ID does not exist"
Nov 03 01:17:37.943009 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.942977    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"]
Nov 03 01:17:37.953526 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:37.953497    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-9btkd"]
Nov 03 01:17:39.242375 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:39.242337    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f474b6f6-e47d-4c13-8f63-b188ca20fce6" path="/var/lib/kubelet/pods/f474b6f6-e47d-4c13-8f63-b188ca20fce6/volumes"
Nov 03 01:17:46.254775 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:46.254744    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f5fd7bcdf-cdr4b_3be4bd66-b870-41a8-931d-94f258522d39/authorino/0.log"
Nov 03 01:17:46.485150 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:46.485108    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-kbvsh_087f9032-547d-4eec-b6fe-b6f692ef2b47/manager/0.log"
Nov 03 01:17:46.684459 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:46.684428    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-wz9nt_8a8544a5-55b2-4894-a924-6c408585fe18/manager/1.log"
Nov 03 01:17:48.041135 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:48.041092    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-6mmrg_56d60aa9-02ef-45c9-be0a-37f090753315/manager/0.log"
Nov 03 01:17:48.670899 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:48.670865    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-wz9nt_8a8544a5-55b2-4894-a924-6c408585fe18/manager/0.log"
Nov 03 01:17:50.760928 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:17:50.760890    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-x5z9p_8c5d5029-6d11-48ee-8771-d2bb8cdcde13/manager/0.log"
Nov 03 01:18:07.362505 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:18:07.362417    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:18:22.692474 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.692429    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf"]
Nov 03 01:18:22.692972 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.692870    5012 cpu_manager.go:414] "containerMap: removing container" podUID="b420bc74-3aaf-4937-9237-9b35dc51ba8e" containerName="payload-pre-processing"
Nov 03 01:18:22.692972 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.692882    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b420bc74-3aaf-4937-9237-9b35dc51ba8e" containerName="payload-pre-processing"
Nov 03 01:18:22.692972 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.692897    5012 cpu_manager.go:414] "containerMap: removing container" podUID="36bbffa3-d72d-4f83-8916-d2d70377d2ea" containerName="payload-processing"
Nov 03 01:18:22.692972 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.692903    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36bbffa3-d72d-4f83-8916-d2d70377d2ea" containerName="payload-processing"
Nov 03 01:18:22.692972 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.692926    5012 cpu_manager.go:414] "containerMap: removing container" podUID="f474b6f6-e47d-4c13-8f63-b188ca20fce6" containerName="istio-proxy"
Nov 03 01:18:22.692972 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.692931    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f474b6f6-e47d-4c13-8f63-b188ca20fce6" containerName="istio-proxy"
Nov 03 01:18:22.692972 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.692945    5012 cpu_manager.go:414] "containerMap: removing container" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main"
Nov 03 01:18:22.692972 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.692950    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main"
Nov 03 01:18:22.692972 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.692967    5012 cpu_manager.go:414] "containerMap: removing container" podUID="d7409925-56ce-416e-b0dc-2a4fde810094" containerName="payload-processing"
Nov 03 01:18:22.692972 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.692972    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7409925-56ce-416e-b0dc-2a4fde810094" containerName="payload-processing"
Nov 03 01:18:22.693338 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.692985    5012 cpu_manager.go:414] "containerMap: removing container" podUID="bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2" containerName="revoke-keys"
Nov 03 01:18:22.693338 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.692990    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2" containerName="revoke-keys"
Nov 03 01:18:22.693338 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.692997    5012 cpu_manager.go:414] "containerMap: removing container" podUID="ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e" containerName="payload-pre-processing"
Nov 03 01:18:22.693338 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.693002    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e" containerName="payload-pre-processing"
Nov 03 01:18:22.693338 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.693076    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7409925-56ce-416e-b0dc-2a4fde810094" containerName="payload-processing"
Nov 03 01:18:22.693338 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.693087    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2" containerName="revoke-keys"
Nov 03 01:18:22.693338 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.693093    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec3b6c1b-27e2-4970-bf0a-8a5685c67d7e" containerName="payload-pre-processing"
Nov 03 01:18:22.693338 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.693100    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="f474b6f6-e47d-4c13-8f63-b188ca20fce6" containerName="istio-proxy"
Nov 03 01:18:22.693338 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.693108    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="b420bc74-3aaf-4937-9237-9b35dc51ba8e" containerName="payload-pre-processing"
Nov 03 01:18:22.693338 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.693113    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="36bbffa3-d72d-4f83-8916-d2d70377d2ea" containerName="payload-processing"
Nov 03 01:18:22.693338 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.693119    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="f474b6f6-e47d-4c13-8f63-b188ca20fce6" containerName="istio-proxy"
Nov 03 01:18:22.693338 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.693126    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba41fa63-4d99-49a5-aa53-5b15efddb4a5" containerName="main"
Nov 03 01:18:22.698000 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.697974    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf"
Nov 03 01:18:22.707522 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.707487    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cprkh\" (UniqueName: \"kubernetes.io/projected/0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f-kube-api-access-cprkh\") pod \"maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf\" (UID: \"0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf"
Nov 03 01:18:22.707691 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.707542    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf\" (UID: \"0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf"
Nov 03 01:18:22.707691 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.707649    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf\" (UID: \"0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf"
Nov 03 01:18:22.719251 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.719194    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf"]
Nov 03 01:18:22.723368 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.723325    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:18:22.723565 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.723396    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:18:22.723565 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.723490    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:18:22.723690 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.723561    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:18:22.808483 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.808436    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf\" (UID: \"0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf"
Nov 03 01:18:22.808695 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.808494    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf\" (UID: \"0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf"
Nov 03 01:18:22.808695 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.808582    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cprkh\" (UniqueName: \"kubernetes.io/projected/0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f-kube-api-access-cprkh\") pod \"maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf\" (UID: \"0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf"
Nov 03 01:18:22.809280 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.809245    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf\" (UID: \"0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf"
Nov 03 01:18:22.811269 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.811244    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf\" (UID: \"0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf"
Nov 03 01:18:22.826088 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:22.826050    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cprkh\" (UniqueName: \"kubernetes.io/projected/0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f-kube-api-access-cprkh\") pod \"maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf\" (UID: \"0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf"
Nov 03 01:18:23.013195 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:23.013104    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf"
Nov 03 01:18:23.310360 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:23.310327    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf"]
Nov 03 01:18:23.312266 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:18:23.312234    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2d3fe4_c9ef_43ed_bf2f_2fbba1d9368f.slice/crio-c7568df54aa5f5011fcec461c711ffa8efeee1db38c7f66d0da6953b5a860337 WatchSource:0}: Error finding container c7568df54aa5f5011fcec461c711ffa8efeee1db38c7f66d0da6953b5a860337: Status 404 returned error can't find the container with id c7568df54aa5f5011fcec461c711ffa8efeee1db38c7f66d0da6953b5a860337
Nov 03 01:18:24.168596 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:24.168559    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf" event={"ID":"0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f","Type":"ContainerStarted","Data":"cff21a24a8c7dea9946bdc5c1167814d6515e045959f3a838bec6c7f4359d116"}
Nov 03 01:18:24.169075 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:24.168615    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf" event={"ID":"0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f","Type":"ContainerStarted","Data":"c7568df54aa5f5011fcec461c711ffa8efeee1db38c7f66d0da6953b5a860337"}
Nov 03 01:18:24.203801 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:24.203747    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf" podStartSLOduration=2.203731099 podStartE2EDuration="2.203731099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:24.202569028 +0000 UTC m=+4387.082153072" watchObservedRunningTime="2026-11-03 01:18:24.203731099 +0000 UTC m=+4387.083315120"
Nov 03 01:18:25.177350 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:25.177314    5012 generic.go:356] "Generic (PLEG): container finished" podID="0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f" containerID="cff21a24a8c7dea9946bdc5c1167814d6515e045959f3a838bec6c7f4359d116" exitCode=0
Nov 03 01:18:25.177791 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:25.177398    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf" event={"ID":"0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f","Type":"ContainerDied","Data":"cff21a24a8c7dea9946bdc5c1167814d6515e045959f3a838bec6c7f4359d116"}
Nov 03 01:18:26.336979 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:26.336941    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf"
Nov 03 01:18:26.339410 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:26.339368    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f-maas-api-openshift-ca\") pod \"0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f\" (UID: \"0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f\") "
Nov 03 01:18:26.339570 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:26.339435    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f-maas-api-xks-ca\") pod \"0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f\" (UID: \"0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f\") "
Nov 03 01:18:26.339570 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:26.339518    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f-kube-api-access-cprkh\" (UniqueName: \"kubernetes.io/projected/0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f-kube-api-access-cprkh\") pod \"0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f\" (UID: \"0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f\") "
Nov 03 01:18:26.339868 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:26.339805    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f-maas-api-openshift-ca" pod "0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f" (UID: "0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:18:26.342136 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:26.342091    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f-maas-api-xks-ca" pod "0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f" (UID: "0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:18:26.342247 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:26.342149    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f-kube-api-access-cprkh" pod "0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f" (UID: "0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f"). InnerVolumeSpecName "kube-api-access-cprkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:18:26.441161 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:26.441060    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cprkh\" (UniqueName: \"kubernetes.io/projected/0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f-kube-api-access-cprkh\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:18:26.441161 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:26.441097    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:18:26.441161 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:26.441107    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:18:27.192958 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:27.192919    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf"
Nov 03 01:18:27.192958 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:27.192927    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf" event={"ID":"0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f","Type":"ContainerDied","Data":"c7568df54aa5f5011fcec461c711ffa8efeee1db38c7f66d0da6953b5a860337"}
Nov 03 01:18:27.193206 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:27.192988    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7568df54aa5f5011fcec461c711ffa8efeee1db38c7f66d0da6953b5a860337"
Nov 03 01:18:33.549731 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:33.549685    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf"]
Nov 03 01:18:33.562590 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:33.562510    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e3bc4e3-mzpzf"]
Nov 03 01:18:35.242465 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:18:35.242418    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f" path="/var/lib/kubelet/pods/0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f/volumes"
Nov 03 01:18:37.362239 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:18:37.362206    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:19:00.589717 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:00.589670    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4hpff"]
Nov 03 01:19:00.590379 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:00.590346    5012 cpu_manager.go:414] "containerMap: removing container" podUID="0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f" containerName="revoke-keys"
Nov 03 01:19:00.590379 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:00.590368    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f" containerName="revoke-keys"
Nov 03 01:19:00.590532 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:00.590386    5012 cpu_manager.go:414] "containerMap: removing container" podUID="f474b6f6-e47d-4c13-8f63-b188ca20fce6" containerName="istio-proxy"
Nov 03 01:19:00.590532 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:00.590395    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f474b6f6-e47d-4c13-8f63-b188ca20fce6" containerName="istio-proxy"
Nov 03 01:19:00.590638 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:00.590555    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2d3fe4-c9ef-43ed-bf2f-2fbba1d9368f" containerName="revoke-keys"
Nov 03 01:19:00.594119 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:00.594097    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4hpff"
Nov 03 01:19:00.594710 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:00.594683    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m2b6\" (UniqueName: \"kubernetes.io/projected/4cf27a36-9d3f-421a-9950-6aec1e21af96-kube-api-access-2m2b6\") pod \"maas-api-revoke-sub-85237e45-4hpff\" (UID: \"4cf27a36-9d3f-421a-9950-6aec1e21af96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4hpff"
Nov 03 01:19:00.594799 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:00.594718    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cf27a36-9d3f-421a-9950-6aec1e21af96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-4hpff\" (UID: \"4cf27a36-9d3f-421a-9950-6aec1e21af96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4hpff"
Nov 03 01:19:00.594893 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:00.594866    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cf27a36-9d3f-421a-9950-6aec1e21af96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-4hpff\" (UID: \"4cf27a36-9d3f-421a-9950-6aec1e21af96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4hpff"
Nov 03 01:19:00.608777 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:00.608748    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:19:00.619338 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:00.619293    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:19:00.622471 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:00.622444    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4hpff"]
Nov 03 01:19:00.673759 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:00.673727    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:19:00.673759 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:00.673745    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:19:00.695554 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:00.695519    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cf27a36-9d3f-421a-9950-6aec1e21af96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-4hpff\" (UID: \"4cf27a36-9d3f-421a-9950-6aec1e21af96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4hpff"
Nov 03 01:19:00.695700 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:00.695579    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m2b6\" (UniqueName: \"kubernetes.io/projected/4cf27a36-9d3f-421a-9950-6aec1e21af96-kube-api-access-2m2b6\") pod \"maas-api-revoke-sub-85237e45-4hpff\" (UID: \"4cf27a36-9d3f-421a-9950-6aec1e21af96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4hpff"
Nov 03 01:19:00.695700 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:00.695616    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cf27a36-9d3f-421a-9950-6aec1e21af96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-4hpff\" (UID: \"4cf27a36-9d3f-421a-9950-6aec1e21af96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4hpff"
Nov 03 01:19:00.696337 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:00.696313    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cf27a36-9d3f-421a-9950-6aec1e21af96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-4hpff\" (UID: \"4cf27a36-9d3f-421a-9950-6aec1e21af96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4hpff"
Nov 03 01:19:00.697869 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:00.697851    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cf27a36-9d3f-421a-9950-6aec1e21af96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-4hpff\" (UID: \"4cf27a36-9d3f-421a-9950-6aec1e21af96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4hpff"
Nov 03 01:19:00.713298 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:00.713272    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m2b6\" (UniqueName: \"kubernetes.io/projected/4cf27a36-9d3f-421a-9950-6aec1e21af96-kube-api-access-2m2b6\") pod \"maas-api-revoke-sub-85237e45-4hpff\" (UID: \"4cf27a36-9d3f-421a-9950-6aec1e21af96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4hpff"
Nov 03 01:19:00.907518 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:00.907483    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4hpff"
Nov 03 01:19:01.202089 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:01.202057    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4hpff"]
Nov 03 01:19:01.204020 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:19:01.203981    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf27a36_9d3f_421a_9950_6aec1e21af96.slice/crio-c2135330607c7d9fa21d7c96b61773f1bc6c7fd0fd2c26aa976d42ddca2c7f02 WatchSource:0}: Error finding container c2135330607c7d9fa21d7c96b61773f1bc6c7fd0fd2c26aa976d42ddca2c7f02: Status 404 returned error can't find the container with id c2135330607c7d9fa21d7c96b61773f1bc6c7fd0fd2c26aa976d42ddca2c7f02
Nov 03 01:19:01.447209 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:01.447170    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4hpff" event={"ID":"4cf27a36-9d3f-421a-9950-6aec1e21af96","Type":"ContainerStarted","Data":"c2135330607c7d9fa21d7c96b61773f1bc6c7fd0fd2c26aa976d42ddca2c7f02"}
Nov 03 01:19:02.455029 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:02.454993    5012 generic.go:356] "Generic (PLEG): container finished" podID="4cf27a36-9d3f-421a-9950-6aec1e21af96" containerID="a8e8800c4774f296ff8527bf14f95409739925ad3778a497161b657745473802" exitCode=0
Nov 03 01:19:02.455408 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:02.455082    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4hpff" event={"ID":"4cf27a36-9d3f-421a-9950-6aec1e21af96","Type":"ContainerDied","Data":"a8e8800c4774f296ff8527bf14f95409739925ad3778a497161b657745473802"}
Nov 03 01:19:03.610383 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:03.610355    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4hpff"
Nov 03 01:19:03.620566 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:03.620537    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cf27a36-9d3f-421a-9950-6aec1e21af96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cf27a36-9d3f-421a-9950-6aec1e21af96-maas-api-openshift-ca\") pod \"4cf27a36-9d3f-421a-9950-6aec1e21af96\" (UID: \"4cf27a36-9d3f-421a-9950-6aec1e21af96\") "
Nov 03 01:19:03.620687 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:03.620581    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cf27a36-9d3f-421a-9950-6aec1e21af96-kube-api-access-2m2b6\" (UniqueName: \"kubernetes.io/projected/4cf27a36-9d3f-421a-9950-6aec1e21af96-kube-api-access-2m2b6\") pod \"4cf27a36-9d3f-421a-9950-6aec1e21af96\" (UID: \"4cf27a36-9d3f-421a-9950-6aec1e21af96\") "
Nov 03 01:19:03.620687 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:03.620620    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cf27a36-9d3f-421a-9950-6aec1e21af96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cf27a36-9d3f-421a-9950-6aec1e21af96-maas-api-xks-ca\") pod \"4cf27a36-9d3f-421a-9950-6aec1e21af96\" (UID: \"4cf27a36-9d3f-421a-9950-6aec1e21af96\") "
Nov 03 01:19:03.620924 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:03.620890    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf27a36-9d3f-421a-9950-6aec1e21af96-maas-api-openshift-ca" pod "4cf27a36-9d3f-421a-9950-6aec1e21af96" (UID: "4cf27a36-9d3f-421a-9950-6aec1e21af96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:19:03.622593 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:03.622573    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf27a36-9d3f-421a-9950-6aec1e21af96-kube-api-access-2m2b6" pod "4cf27a36-9d3f-421a-9950-6aec1e21af96" (UID: "4cf27a36-9d3f-421a-9950-6aec1e21af96"). InnerVolumeSpecName "kube-api-access-2m2b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:19:03.622676 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:03.622607    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cf27a36-9d3f-421a-9950-6aec1e21af96-maas-api-xks-ca" pod "4cf27a36-9d3f-421a-9950-6aec1e21af96" (UID: "4cf27a36-9d3f-421a-9950-6aec1e21af96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:19:03.721463 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:03.721400    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cf27a36-9d3f-421a-9950-6aec1e21af96-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:19:03.721463 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:03.721428    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m2b6\" (UniqueName: \"kubernetes.io/projected/4cf27a36-9d3f-421a-9950-6aec1e21af96-kube-api-access-2m2b6\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:19:03.721463 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:03.721437    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cf27a36-9d3f-421a-9950-6aec1e21af96-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:19:04.467527 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:04.467487    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4hpff" event={"ID":"4cf27a36-9d3f-421a-9950-6aec1e21af96","Type":"ContainerDied","Data":"c2135330607c7d9fa21d7c96b61773f1bc6c7fd0fd2c26aa976d42ddca2c7f02"}
Nov 03 01:19:04.467527 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:04.467535    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2135330607c7d9fa21d7c96b61773f1bc6c7fd0fd2c26aa976d42ddca2c7f02"
Nov 03 01:19:04.467751 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:04.467498    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4hpff"
Nov 03 01:19:07.361388 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:19:07.361353    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:19:13.551254 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.551218    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"]
Nov 03 01:19:13.551982 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.551725    5012 cpu_manager.go:414] "containerMap: removing container" podUID="4cf27a36-9d3f-421a-9950-6aec1e21af96" containerName="revoke-keys"
Nov 03 01:19:13.551982 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.551747    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cf27a36-9d3f-421a-9950-6aec1e21af96" containerName="revoke-keys"
Nov 03 01:19:13.551982 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.551836    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cf27a36-9d3f-421a-9950-6aec1e21af96" containerName="revoke-keys"
Nov 03 01:19:13.555411 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.555391    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.585108 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.585083    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"]
Nov 03 01:19:13.603191 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.603164    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv4fd\" (UniqueName: \"kubernetes.io/projected/c2788e33-32b9-47f6-8d84-d8663aa07a99-kube-api-access-pv4fd\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.603309 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.603209    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2788e33-32b9-47f6-8d84-d8663aa07a99-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.603360 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.603317    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2788e33-32b9-47f6-8d84-d8663aa07a99-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.603360 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.603355    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2788e33-32b9-47f6-8d84-d8663aa07a99-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.603434 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.603373    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2788e33-32b9-47f6-8d84-d8663aa07a99-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.603481 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.603462    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2788e33-32b9-47f6-8d84-d8663aa07a99-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.603539 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.603523    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2788e33-32b9-47f6-8d84-d8663aa07a99-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.603582 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.603568    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2788e33-32b9-47f6-8d84-d8663aa07a99-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.603701 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.603675    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2788e33-32b9-47f6-8d84-d8663aa07a99-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.705186 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.705148    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2788e33-32b9-47f6-8d84-d8663aa07a99-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.705326 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.705190    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2788e33-32b9-47f6-8d84-d8663aa07a99-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.705326 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.705218    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2788e33-32b9-47f6-8d84-d8663aa07a99-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.705326 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.705256    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2788e33-32b9-47f6-8d84-d8663aa07a99-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.705326 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.705284    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2788e33-32b9-47f6-8d84-d8663aa07a99-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.705326 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.705314    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2788e33-32b9-47f6-8d84-d8663aa07a99-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.705605 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.705540    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2788e33-32b9-47f6-8d84-d8663aa07a99-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.705666 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.705615    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv4fd\" (UniqueName: \"kubernetes.io/projected/c2788e33-32b9-47f6-8d84-d8663aa07a99-kube-api-access-pv4fd\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.705666 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.705646    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2788e33-32b9-47f6-8d84-d8663aa07a99-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.705794 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.705747    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2788e33-32b9-47f6-8d84-d8663aa07a99-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.705852 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.705794    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2788e33-32b9-47f6-8d84-d8663aa07a99-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.706017 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.705998    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2788e33-32b9-47f6-8d84-d8663aa07a99-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.706104 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.706031    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2788e33-32b9-47f6-8d84-d8663aa07a99-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.706182 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.706157    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2788e33-32b9-47f6-8d84-d8663aa07a99-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.707632 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.707610    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2788e33-32b9-47f6-8d84-d8663aa07a99-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.708038 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.708020    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2788e33-32b9-47f6-8d84-d8663aa07a99-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.733870 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.733844    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2788e33-32b9-47f6-8d84-d8663aa07a99-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.734103 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.734086    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv4fd\" (UniqueName: \"kubernetes.io/projected/c2788e33-32b9-47f6-8d84-d8663aa07a99-kube-api-access-pv4fd\") pod \"maas-default-gateway-openshift-default-6cc458d887-hstf4\" (UID: \"c2788e33-32b9-47f6-8d84-d8663aa07a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:13.871026 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:13.870999    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:14.190243 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:14.190214    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"]
Nov 03 01:19:14.191945 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:19:14.191888    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2788e33_32b9_47f6_8d84_d8663aa07a99.slice/crio-7bd7c2c272f2261c9eb10da6e04147c845a3391a937702618b4362d3ce8a991d WatchSource:0}: Error finding container 7bd7c2c272f2261c9eb10da6e04147c845a3391a937702618b4362d3ce8a991d: Status 404 returned error can't find the container with id 7bd7c2c272f2261c9eb10da6e04147c845a3391a937702618b4362d3ce8a991d
Nov 03 01:19:14.194148 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:14.194111    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:19:14.194259 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:14.194178    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:19:14.194259 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:14.194209    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:19:14.531417 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:14.531316    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4" event={"ID":"c2788e33-32b9-47f6-8d84-d8663aa07a99","Type":"ContainerStarted","Data":"539a5878c3ee1f52618a34cde40cd7b356b647ba545940c9058b8f1bd624813a"}
Nov 03 01:19:14.531417 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:14.531381    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4" event={"ID":"c2788e33-32b9-47f6-8d84-d8663aa07a99","Type":"ContainerStarted","Data":"7bd7c2c272f2261c9eb10da6e04147c845a3391a937702618b4362d3ce8a991d"}
Nov 03 01:19:14.574660 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:14.574603    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4" podStartSLOduration=1.5745890710000001 podStartE2EDuration="1.574589071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:19:14.57066125 +0000 UTC m=+4437.450245296" watchObservedRunningTime="2026-11-03 01:19:14.574589071 +0000 UTC m=+4437.454173114"
Nov 03 01:19:14.871150 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:14.871114    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:15.873373 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:15.873341    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hstf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Nov 03 01:19:15.873798 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:15.873391    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4" podUID="c2788e33-32b9-47f6-8d84-d8663aa07a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Nov 03 01:19:16.871309 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:16.871271    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hstf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Nov 03 01:19:16.871567 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:16.871327    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4" podUID="c2788e33-32b9-47f6-8d84-d8663aa07a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Nov 03 01:19:17.872140 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:17.872104    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hstf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Nov 03 01:19:17.872579 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:17.872160    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4" podUID="c2788e33-32b9-47f6-8d84-d8663aa07a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Nov 03 01:19:18.872151 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:18.872113    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hstf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Nov 03 01:19:18.872582 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:18.872177    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4" podUID="c2788e33-32b9-47f6-8d84-d8663aa07a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Nov 03 01:19:19.872111 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:19.872072    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hstf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Nov 03 01:19:19.872291 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:19.872133    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4" podUID="c2788e33-32b9-47f6-8d84-d8663aa07a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Nov 03 01:19:20.871531 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:20.871497    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hstf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Nov 03 01:19:20.871711 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:20.871554    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4" podUID="c2788e33-32b9-47f6-8d84-d8663aa07a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Nov 03 01:19:22.537780 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:22.537744    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:22.538236 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:22.538200    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:22.539095 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:22.539068    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4"
Nov 03 01:19:37.369009 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:19:37.368887    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:19:41.190640 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:41.190601    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xgg58"]
Nov 03 01:19:41.194554 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:41.194535    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xgg58"
Nov 03 01:19:41.209756 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:41.209733    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:19:41.213406 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:41.213381    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xgg58"]
Nov 03 01:19:41.220575 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:41.220551    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:19:41.220747 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:41.220726    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:19:41.220851 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:41.220809    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:19:41.298324 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:41.298292    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8pcn\" (UniqueName: \"kubernetes.io/projected/5b78873b-8271-4452-9b32-f761dd4d7285-kube-api-access-r8pcn\") pod \"maas-api-revoke-sub-f904bff9-xgg58\" (UID: \"5b78873b-8271-4452-9b32-f761dd4d7285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xgg58"
Nov 03 01:19:41.298508 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:41.298364    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b78873b-8271-4452-9b32-f761dd4d7285-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-xgg58\" (UID: \"5b78873b-8271-4452-9b32-f761dd4d7285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xgg58"
Nov 03 01:19:41.298508 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:41.298461    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b78873b-8271-4452-9b32-f761dd4d7285-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-xgg58\" (UID: \"5b78873b-8271-4452-9b32-f761dd4d7285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xgg58"
Nov 03 01:19:41.399804 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:41.399767    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b78873b-8271-4452-9b32-f761dd4d7285-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-xgg58\" (UID: \"5b78873b-8271-4452-9b32-f761dd4d7285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xgg58"
Nov 03 01:19:41.399804 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:41.399806    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b78873b-8271-4452-9b32-f761dd4d7285-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-xgg58\" (UID: \"5b78873b-8271-4452-9b32-f761dd4d7285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xgg58"
Nov 03 01:19:41.400064 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:41.399874    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8pcn\" (UniqueName: \"kubernetes.io/projected/5b78873b-8271-4452-9b32-f761dd4d7285-kube-api-access-r8pcn\") pod \"maas-api-revoke-sub-f904bff9-xgg58\" (UID: \"5b78873b-8271-4452-9b32-f761dd4d7285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xgg58"
Nov 03 01:19:41.400508 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:41.400474    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b78873b-8271-4452-9b32-f761dd4d7285-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-xgg58\" (UID: \"5b78873b-8271-4452-9b32-f761dd4d7285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xgg58"
Nov 03 01:19:41.402173 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:41.402151    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b78873b-8271-4452-9b32-f761dd4d7285-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-xgg58\" (UID: \"5b78873b-8271-4452-9b32-f761dd4d7285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xgg58"
Nov 03 01:19:41.416560 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:41.416537    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8pcn\" (UniqueName: \"kubernetes.io/projected/5b78873b-8271-4452-9b32-f761dd4d7285-kube-api-access-r8pcn\") pod \"maas-api-revoke-sub-f904bff9-xgg58\" (UID: \"5b78873b-8271-4452-9b32-f761dd4d7285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xgg58"
Nov 03 01:19:41.507688 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:41.507612    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xgg58"
Nov 03 01:19:41.799658 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:41.799629    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xgg58"]
Nov 03 01:19:41.801458 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:19:41.801420    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b78873b_8271_4452_9b32_f761dd4d7285.slice/crio-e246d0a5b5cfc095825037e8c863bb9808fbd4244a6c24a38a6e728b9b9812b6 WatchSource:0}: Error finding container e246d0a5b5cfc095825037e8c863bb9808fbd4244a6c24a38a6e728b9b9812b6: Status 404 returned error can't find the container with id e246d0a5b5cfc095825037e8c863bb9808fbd4244a6c24a38a6e728b9b9812b6
Nov 03 01:19:42.713547 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:42.713520    5012 generic.go:356] "Generic (PLEG): container finished" podID="5b78873b-8271-4452-9b32-f761dd4d7285" containerID="42a6fae3fa6a8f50a3cdcce879d1345dc026e3d5126fcddc081cf55cf2613b9d" exitCode=0
Nov 03 01:19:42.713875 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:42.713598    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xgg58" event={"ID":"5b78873b-8271-4452-9b32-f761dd4d7285","Type":"ContainerDied","Data":"42a6fae3fa6a8f50a3cdcce879d1345dc026e3d5126fcddc081cf55cf2613b9d"}
Nov 03 01:19:42.713875 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:42.713656    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xgg58" event={"ID":"5b78873b-8271-4452-9b32-f761dd4d7285","Type":"ContainerStarted","Data":"e246d0a5b5cfc095825037e8c863bb9808fbd4244a6c24a38a6e728b9b9812b6"}
Nov 03 01:19:43.865059 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:43.865036    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xgg58"
Nov 03 01:19:43.923317 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:43.923288    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b78873b-8271-4452-9b32-f761dd4d7285-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b78873b-8271-4452-9b32-f761dd4d7285-maas-api-xks-ca\") pod \"5b78873b-8271-4452-9b32-f761dd4d7285\" (UID: \"5b78873b-8271-4452-9b32-f761dd4d7285\") "
Nov 03 01:19:43.923483 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:43.923349    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b78873b-8271-4452-9b32-f761dd4d7285-kube-api-access-r8pcn\" (UniqueName: \"kubernetes.io/projected/5b78873b-8271-4452-9b32-f761dd4d7285-kube-api-access-r8pcn\") pod \"5b78873b-8271-4452-9b32-f761dd4d7285\" (UID: \"5b78873b-8271-4452-9b32-f761dd4d7285\") "
Nov 03 01:19:43.923483 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:43.923376    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b78873b-8271-4452-9b32-f761dd4d7285-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b78873b-8271-4452-9b32-f761dd4d7285-maas-api-openshift-ca\") pod \"5b78873b-8271-4452-9b32-f761dd4d7285\" (UID: \"5b78873b-8271-4452-9b32-f761dd4d7285\") "
Nov 03 01:19:43.923871 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:43.923841    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b78873b-8271-4452-9b32-f761dd4d7285-maas-api-openshift-ca" pod "5b78873b-8271-4452-9b32-f761dd4d7285" (UID: "5b78873b-8271-4452-9b32-f761dd4d7285"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:19:43.925415 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:43.925389    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b78873b-8271-4452-9b32-f761dd4d7285-maas-api-xks-ca" pod "5b78873b-8271-4452-9b32-f761dd4d7285" (UID: "5b78873b-8271-4452-9b32-f761dd4d7285"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:19:43.925539 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:43.925479    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b78873b-8271-4452-9b32-f761dd4d7285-kube-api-access-r8pcn" pod "5b78873b-8271-4452-9b32-f761dd4d7285" (UID: "5b78873b-8271-4452-9b32-f761dd4d7285"). InnerVolumeSpecName "kube-api-access-r8pcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:19:44.024551 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:44.024475    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b78873b-8271-4452-9b32-f761dd4d7285-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:19:44.024551 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:44.024505    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8pcn\" (UniqueName: \"kubernetes.io/projected/5b78873b-8271-4452-9b32-f761dd4d7285-kube-api-access-r8pcn\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:19:44.024551 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:44.024516    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b78873b-8271-4452-9b32-f761dd4d7285-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:19:44.727138 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:44.727108    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xgg58"
Nov 03 01:19:44.727458 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:44.727136    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xgg58" event={"ID":"5b78873b-8271-4452-9b32-f761dd4d7285","Type":"ContainerDied","Data":"e246d0a5b5cfc095825037e8c863bb9808fbd4244a6c24a38a6e728b9b9812b6"}
Nov 03 01:19:44.727458 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:19:44.727179    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e246d0a5b5cfc095825037e8c863bb9808fbd4244a6c24a38a6e728b9b9812b6"
Nov 03 01:20:05.411355 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:05.411321    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2zzwf"]
Nov 03 01:20:05.411755 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:05.411742    5012 cpu_manager.go:414] "containerMap: removing container" podUID="5b78873b-8271-4452-9b32-f761dd4d7285" containerName="revoke-keys"
Nov 03 01:20:05.411806 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:05.411756    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b78873b-8271-4452-9b32-f761dd4d7285" containerName="revoke-keys"
Nov 03 01:20:05.411848 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:05.411820    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b78873b-8271-4452-9b32-f761dd4d7285" containerName="revoke-keys"
Nov 03 01:20:05.415720 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:05.415684    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2zzwf"
Nov 03 01:20:05.430018 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:05.429993    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:20:05.433682 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:05.433656    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2zzwf"]
Nov 03 01:20:05.440424 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:05.440402    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:20:05.440657 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:05.440639    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:20:05.440738 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:05.440660    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:20:05.481316 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:05.481291    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faaf71e2-e4b4-4763-8899-c21ded229b62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-2zzwf\" (UID: \"faaf71e2-e4b4-4763-8899-c21ded229b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2zzwf"
Nov 03 01:20:05.481413 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:05.481339    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr59n\" (UniqueName: \"kubernetes.io/projected/faaf71e2-e4b4-4763-8899-c21ded229b62-kube-api-access-vr59n\") pod \"maas-api-revoke-sub-f03629db-2zzwf\" (UID: \"faaf71e2-e4b4-4763-8899-c21ded229b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2zzwf"
Nov 03 01:20:05.481463 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:05.481427    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faaf71e2-e4b4-4763-8899-c21ded229b62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-2zzwf\" (UID: \"faaf71e2-e4b4-4763-8899-c21ded229b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2zzwf"
Nov 03 01:20:05.581827 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:05.581798    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faaf71e2-e4b4-4763-8899-c21ded229b62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-2zzwf\" (UID: \"faaf71e2-e4b4-4763-8899-c21ded229b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2zzwf"
Nov 03 01:20:05.581955 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:05.581845    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr59n\" (UniqueName: \"kubernetes.io/projected/faaf71e2-e4b4-4763-8899-c21ded229b62-kube-api-access-vr59n\") pod \"maas-api-revoke-sub-f03629db-2zzwf\" (UID: \"faaf71e2-e4b4-4763-8899-c21ded229b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2zzwf"
Nov 03 01:20:05.581955 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:05.581874    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faaf71e2-e4b4-4763-8899-c21ded229b62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-2zzwf\" (UID: \"faaf71e2-e4b4-4763-8899-c21ded229b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2zzwf"
Nov 03 01:20:05.582480 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:05.582457    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faaf71e2-e4b4-4763-8899-c21ded229b62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-2zzwf\" (UID: \"faaf71e2-e4b4-4763-8899-c21ded229b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2zzwf"
Nov 03 01:20:05.584208 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:05.584187    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faaf71e2-e4b4-4763-8899-c21ded229b62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-2zzwf\" (UID: \"faaf71e2-e4b4-4763-8899-c21ded229b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2zzwf"
Nov 03 01:20:05.599206 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:05.599178    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr59n\" (UniqueName: \"kubernetes.io/projected/faaf71e2-e4b4-4763-8899-c21ded229b62-kube-api-access-vr59n\") pod \"maas-api-revoke-sub-f03629db-2zzwf\" (UID: \"faaf71e2-e4b4-4763-8899-c21ded229b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2zzwf"
Nov 03 01:20:05.730944 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:05.730863    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2zzwf"
Nov 03 01:20:06.022813 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:06.022787    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2zzwf"]
Nov 03 01:20:06.024369 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:20:06.024340    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaaf71e2_e4b4_4763_8899_c21ded229b62.slice/crio-7937ad181e691aef6ad229a50b784b7a4732e1af7aba05c8a5c4a450dfb63532 WatchSource:0}: Error finding container 7937ad181e691aef6ad229a50b784b7a4732e1af7aba05c8a5c4a450dfb63532: Status 404 returned error can't find the container with id 7937ad181e691aef6ad229a50b784b7a4732e1af7aba05c8a5c4a450dfb63532
Nov 03 01:20:06.957925 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:06.957880    5012 generic.go:356] "Generic (PLEG): container finished" podID="faaf71e2-e4b4-4763-8899-c21ded229b62" containerID="e6032ff8afbd651dc2e0eb6b5b874c052bd74bb6ff148fdf37d5c8cf07c1729c" exitCode=0
Nov 03 01:20:06.958299 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:06.957969    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2zzwf" event={"ID":"faaf71e2-e4b4-4763-8899-c21ded229b62","Type":"ContainerDied","Data":"e6032ff8afbd651dc2e0eb6b5b874c052bd74bb6ff148fdf37d5c8cf07c1729c"}
Nov 03 01:20:06.958299 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:06.958022    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2zzwf" event={"ID":"faaf71e2-e4b4-4763-8899-c21ded229b62","Type":"ContainerStarted","Data":"7937ad181e691aef6ad229a50b784b7a4732e1af7aba05c8a5c4a450dfb63532"}
Nov 03 01:20:07.362514 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:20:07.362478    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:20:08.102176 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:08.102151    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2zzwf"
Nov 03 01:20:08.205943 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:08.205893    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faaf71e2-e4b4-4763-8899-c21ded229b62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faaf71e2-e4b4-4763-8899-c21ded229b62-maas-api-xks-ca\") pod \"faaf71e2-e4b4-4763-8899-c21ded229b62\" (UID: \"faaf71e2-e4b4-4763-8899-c21ded229b62\") "
Nov 03 01:20:08.206093 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:08.205958    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faaf71e2-e4b4-4763-8899-c21ded229b62-kube-api-access-vr59n\" (UniqueName: \"kubernetes.io/projected/faaf71e2-e4b4-4763-8899-c21ded229b62-kube-api-access-vr59n\") pod \"faaf71e2-e4b4-4763-8899-c21ded229b62\" (UID: \"faaf71e2-e4b4-4763-8899-c21ded229b62\") "
Nov 03 01:20:08.206093 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:08.206054    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faaf71e2-e4b4-4763-8899-c21ded229b62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faaf71e2-e4b4-4763-8899-c21ded229b62-maas-api-openshift-ca\") pod \"faaf71e2-e4b4-4763-8899-c21ded229b62\" (UID: \"faaf71e2-e4b4-4763-8899-c21ded229b62\") "
Nov 03 01:20:08.206545 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:08.206514    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faaf71e2-e4b4-4763-8899-c21ded229b62-maas-api-openshift-ca" pod "faaf71e2-e4b4-4763-8899-c21ded229b62" (UID: "faaf71e2-e4b4-4763-8899-c21ded229b62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:20:08.207967 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:08.207901    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faaf71e2-e4b4-4763-8899-c21ded229b62-maas-api-xks-ca" pod "faaf71e2-e4b4-4763-8899-c21ded229b62" (UID: "faaf71e2-e4b4-4763-8899-c21ded229b62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:20:08.208079 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:08.207988    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faaf71e2-e4b4-4763-8899-c21ded229b62-kube-api-access-vr59n" pod "faaf71e2-e4b4-4763-8899-c21ded229b62" (UID: "faaf71e2-e4b4-4763-8899-c21ded229b62"). InnerVolumeSpecName "kube-api-access-vr59n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:20:08.307364 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:08.307339    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faaf71e2-e4b4-4763-8899-c21ded229b62-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:20:08.307478 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:08.307367    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faaf71e2-e4b4-4763-8899-c21ded229b62-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:20:08.307478 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:08.307382    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr59n\" (UniqueName: \"kubernetes.io/projected/faaf71e2-e4b4-4763-8899-c21ded229b62-kube-api-access-vr59n\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:20:08.973238 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:08.973158    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2zzwf"
Nov 03 01:20:08.973238 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:08.973200    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2zzwf" event={"ID":"faaf71e2-e4b4-4763-8899-c21ded229b62","Type":"ContainerDied","Data":"7937ad181e691aef6ad229a50b784b7a4732e1af7aba05c8a5c4a450dfb63532"}
Nov 03 01:20:08.973445 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:08.973257    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7937ad181e691aef6ad229a50b784b7a4732e1af7aba05c8a5c4a450dfb63532"
Nov 03 01:20:18.675363 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:18.675317    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 01:20:18.679297 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:18.679271    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 01:20:18.680515 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:18.680491    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 01:20:18.685243 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:18.685219    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 01:20:21.637828 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:21.637788    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x6v55"]
Nov 03 01:20:21.638245 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:21.638233    5012 cpu_manager.go:414] "containerMap: removing container" podUID="faaf71e2-e4b4-4763-8899-c21ded229b62" containerName="revoke-keys"
Nov 03 01:20:21.638293 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:21.638246    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faaf71e2-e4b4-4763-8899-c21ded229b62" containerName="revoke-keys"
Nov 03 01:20:21.638346 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:21.638335    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="faaf71e2-e4b4-4763-8899-c21ded229b62" containerName="revoke-keys"
Nov 03 01:20:21.641894 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:21.641876    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x6v55"
Nov 03 01:20:21.665978 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:21.665954    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:20:21.666187 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:21.666033    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:20:21.666282 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:21.666073    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:20:21.666282 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:21.666140    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:20:21.670789 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:21.670765    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x6v55"]
Nov 03 01:20:21.776936 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:21.776883    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f34a890-46ff-4ab9-a5be-b7ba879cc7fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-x6v55\" (UID: \"5f34a890-46ff-4ab9-a5be-b7ba879cc7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x6v55"
Nov 03 01:20:21.777075 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:21.776974    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv6rl\" (UniqueName: \"kubernetes.io/projected/5f34a890-46ff-4ab9-a5be-b7ba879cc7fa-kube-api-access-hv6rl\") pod \"maas-api-revoke-sub-0165d7d2-x6v55\" (UID: \"5f34a890-46ff-4ab9-a5be-b7ba879cc7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x6v55"
Nov 03 01:20:21.777075 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:21.777054    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f34a890-46ff-4ab9-a5be-b7ba879cc7fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-x6v55\" (UID: \"5f34a890-46ff-4ab9-a5be-b7ba879cc7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x6v55"
Nov 03 01:20:21.878277 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:21.878249    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f34a890-46ff-4ab9-a5be-b7ba879cc7fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-x6v55\" (UID: \"5f34a890-46ff-4ab9-a5be-b7ba879cc7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x6v55"
Nov 03 01:20:21.878388 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:21.878346    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f34a890-46ff-4ab9-a5be-b7ba879cc7fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-x6v55\" (UID: \"5f34a890-46ff-4ab9-a5be-b7ba879cc7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x6v55"
Nov 03 01:20:21.878459 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:21.878386    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv6rl\" (UniqueName: \"kubernetes.io/projected/5f34a890-46ff-4ab9-a5be-b7ba879cc7fa-kube-api-access-hv6rl\") pod \"maas-api-revoke-sub-0165d7d2-x6v55\" (UID: \"5f34a890-46ff-4ab9-a5be-b7ba879cc7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x6v55"
Nov 03 01:20:21.879046 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:21.879022    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f34a890-46ff-4ab9-a5be-b7ba879cc7fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-x6v55\" (UID: \"5f34a890-46ff-4ab9-a5be-b7ba879cc7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x6v55"
Nov 03 01:20:21.880642 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:21.880624    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f34a890-46ff-4ab9-a5be-b7ba879cc7fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-x6v55\" (UID: \"5f34a890-46ff-4ab9-a5be-b7ba879cc7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x6v55"
Nov 03 01:20:21.899240 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:21.899187    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv6rl\" (UniqueName: \"kubernetes.io/projected/5f34a890-46ff-4ab9-a5be-b7ba879cc7fa-kube-api-access-hv6rl\") pod \"maas-api-revoke-sub-0165d7d2-x6v55\" (UID: \"5f34a890-46ff-4ab9-a5be-b7ba879cc7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x6v55"
Nov 03 01:20:21.954783 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:21.954743    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x6v55"
Nov 03 01:20:22.249128 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:22.249093    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x6v55"]
Nov 03 01:20:22.249683 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:20:22.249657    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f34a890_46ff_4ab9_a5be_b7ba879cc7fa.slice/crio-fa548ce7e54a8c544d105547d6e41c900d2a6de20246d0d8dad61d9a9c972991 WatchSource:0}: Error finding container fa548ce7e54a8c544d105547d6e41c900d2a6de20246d0d8dad61d9a9c972991: Status 404 returned error can't find the container with id fa548ce7e54a8c544d105547d6e41c900d2a6de20246d0d8dad61d9a9c972991
Nov 03 01:20:23.077784 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:23.077753    5012 generic.go:356] "Generic (PLEG): container finished" podID="5f34a890-46ff-4ab9-a5be-b7ba879cc7fa" containerID="d36d2177bc42e4fb6f2479e1f9293af782eeddec756522dbb648e5d1ae7db8c2" exitCode=0
Nov 03 01:20:23.078129 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:23.077836    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x6v55" event={"ID":"5f34a890-46ff-4ab9-a5be-b7ba879cc7fa","Type":"ContainerDied","Data":"d36d2177bc42e4fb6f2479e1f9293af782eeddec756522dbb648e5d1ae7db8c2"}
Nov 03 01:20:23.078129 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:23.077888    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x6v55" event={"ID":"5f34a890-46ff-4ab9-a5be-b7ba879cc7fa","Type":"ContainerStarted","Data":"fa548ce7e54a8c544d105547d6e41c900d2a6de20246d0d8dad61d9a9c972991"}
Nov 03 01:20:24.230761 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:24.230734    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x6v55"
Nov 03 01:20:24.297760 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:24.297732    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f34a890-46ff-4ab9-a5be-b7ba879cc7fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f34a890-46ff-4ab9-a5be-b7ba879cc7fa-maas-api-xks-ca\") pod \"5f34a890-46ff-4ab9-a5be-b7ba879cc7fa\" (UID: \"5f34a890-46ff-4ab9-a5be-b7ba879cc7fa\") "
Nov 03 01:20:24.297950 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:24.297814    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f34a890-46ff-4ab9-a5be-b7ba879cc7fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f34a890-46ff-4ab9-a5be-b7ba879cc7fa-maas-api-openshift-ca\") pod \"5f34a890-46ff-4ab9-a5be-b7ba879cc7fa\" (UID: \"5f34a890-46ff-4ab9-a5be-b7ba879cc7fa\") "
Nov 03 01:20:24.297950 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:24.297946    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f34a890-46ff-4ab9-a5be-b7ba879cc7fa-kube-api-access-hv6rl\" (UniqueName: \"kubernetes.io/projected/5f34a890-46ff-4ab9-a5be-b7ba879cc7fa-kube-api-access-hv6rl\") pod \"5f34a890-46ff-4ab9-a5be-b7ba879cc7fa\" (UID: \"5f34a890-46ff-4ab9-a5be-b7ba879cc7fa\") "
Nov 03 01:20:24.298292 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:24.298260    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f34a890-46ff-4ab9-a5be-b7ba879cc7fa-maas-api-openshift-ca" pod "5f34a890-46ff-4ab9-a5be-b7ba879cc7fa" (UID: "5f34a890-46ff-4ab9-a5be-b7ba879cc7fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:20:24.299747 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:24.299727    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f34a890-46ff-4ab9-a5be-b7ba879cc7fa-maas-api-xks-ca" pod "5f34a890-46ff-4ab9-a5be-b7ba879cc7fa" (UID: "5f34a890-46ff-4ab9-a5be-b7ba879cc7fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:20:24.299894 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:24.299873    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f34a890-46ff-4ab9-a5be-b7ba879cc7fa-kube-api-access-hv6rl" pod "5f34a890-46ff-4ab9-a5be-b7ba879cc7fa" (UID: "5f34a890-46ff-4ab9-a5be-b7ba879cc7fa"). InnerVolumeSpecName "kube-api-access-hv6rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:20:24.398757 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:24.398729    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f34a890-46ff-4ab9-a5be-b7ba879cc7fa-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:20:24.398757 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:24.398756    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv6rl\" (UniqueName: \"kubernetes.io/projected/5f34a890-46ff-4ab9-a5be-b7ba879cc7fa-kube-api-access-hv6rl\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:20:24.398988 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:24.398772    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f34a890-46ff-4ab9-a5be-b7ba879cc7fa-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:20:25.091051 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:25.091002    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x6v55" event={"ID":"5f34a890-46ff-4ab9-a5be-b7ba879cc7fa","Type":"ContainerDied","Data":"fa548ce7e54a8c544d105547d6e41c900d2a6de20246d0d8dad61d9a9c972991"}
Nov 03 01:20:25.091261 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:25.091066    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa548ce7e54a8c544d105547d6e41c900d2a6de20246d0d8dad61d9a9c972991"
Nov 03 01:20:25.091261 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:25.091026    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x6v55"
Nov 03 01:20:32.691610 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:32.691574    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cnz49"]
Nov 03 01:20:32.692057 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:32.692034    5012 cpu_manager.go:414] "containerMap: removing container" podUID="5f34a890-46ff-4ab9-a5be-b7ba879cc7fa" containerName="revoke-keys"
Nov 03 01:20:32.692057 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:32.692049    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f34a890-46ff-4ab9-a5be-b7ba879cc7fa" containerName="revoke-keys"
Nov 03 01:20:32.692158 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:32.692122    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f34a890-46ff-4ab9-a5be-b7ba879cc7fa" containerName="revoke-keys"
Nov 03 01:20:32.695264 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:32.695238    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cnz49"
Nov 03 01:20:32.714292 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:32.714262    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cnz49"]
Nov 03 01:20:32.720697 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:32.720669    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:20:32.720818 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:32.720724    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:20:32.720818 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:32.720727    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:20:32.720818 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:32.720777    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:20:32.875220 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:32.875189    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cafd1877-5d84-4f0a-aa25-a26378cd2732-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-cnz49\" (UID: \"cafd1877-5d84-4f0a-aa25-a26378cd2732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cnz49"
Nov 03 01:20:32.875354 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:32.875226    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p9md\" (UniqueName: \"kubernetes.io/projected/cafd1877-5d84-4f0a-aa25-a26378cd2732-kube-api-access-6p9md\") pod \"maas-api-revoke-sub-6b19338f-cnz49\" (UID: \"cafd1877-5d84-4f0a-aa25-a26378cd2732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cnz49"
Nov 03 01:20:32.875354 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:32.875249    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cafd1877-5d84-4f0a-aa25-a26378cd2732-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-cnz49\" (UID: \"cafd1877-5d84-4f0a-aa25-a26378cd2732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cnz49"
Nov 03 01:20:32.976606 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:32.976537    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cafd1877-5d84-4f0a-aa25-a26378cd2732-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-cnz49\" (UID: \"cafd1877-5d84-4f0a-aa25-a26378cd2732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cnz49"
Nov 03 01:20:32.976606 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:32.976574    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p9md\" (UniqueName: \"kubernetes.io/projected/cafd1877-5d84-4f0a-aa25-a26378cd2732-kube-api-access-6p9md\") pod \"maas-api-revoke-sub-6b19338f-cnz49\" (UID: \"cafd1877-5d84-4f0a-aa25-a26378cd2732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cnz49"
Nov 03 01:20:32.976606 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:32.976604    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cafd1877-5d84-4f0a-aa25-a26378cd2732-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-cnz49\" (UID: \"cafd1877-5d84-4f0a-aa25-a26378cd2732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cnz49"
Nov 03 01:20:32.977380 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:32.977355    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cafd1877-5d84-4f0a-aa25-a26378cd2732-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-cnz49\" (UID: \"cafd1877-5d84-4f0a-aa25-a26378cd2732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cnz49"
Nov 03 01:20:32.978968 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:32.978943    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cafd1877-5d84-4f0a-aa25-a26378cd2732-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-cnz49\" (UID: \"cafd1877-5d84-4f0a-aa25-a26378cd2732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cnz49"
Nov 03 01:20:32.994218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:32.994193    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p9md\" (UniqueName: \"kubernetes.io/projected/cafd1877-5d84-4f0a-aa25-a26378cd2732-kube-api-access-6p9md\") pod \"maas-api-revoke-sub-6b19338f-cnz49\" (UID: \"cafd1877-5d84-4f0a-aa25-a26378cd2732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cnz49"
Nov 03 01:20:33.008076 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:33.008054    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cnz49"
Nov 03 01:20:33.300171 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:33.300138    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cnz49"]
Nov 03 01:20:33.301956 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:20:33.301924    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcafd1877_5d84_4f0a_aa25_a26378cd2732.slice/crio-d0fe18eb233250ade83c5b0da785dec4e4de3ef158e38d99baa094420fa22381 WatchSource:0}: Error finding container d0fe18eb233250ade83c5b0da785dec4e4de3ef158e38d99baa094420fa22381: Status 404 returned error can't find the container with id d0fe18eb233250ade83c5b0da785dec4e4de3ef158e38d99baa094420fa22381
Nov 03 01:20:34.153263 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:34.153237    5012 generic.go:356] "Generic (PLEG): container finished" podID="cafd1877-5d84-4f0a-aa25-a26378cd2732" containerID="e04713d3495423d2e4a61196dbfbb872bc9704c1db60b2b99f40f61124bcc18b" exitCode=0
Nov 03 01:20:34.153586 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:34.153315    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cnz49" event={"ID":"cafd1877-5d84-4f0a-aa25-a26378cd2732","Type":"ContainerDied","Data":"e04713d3495423d2e4a61196dbfbb872bc9704c1db60b2b99f40f61124bcc18b"}
Nov 03 01:20:34.153586 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:34.153372    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cnz49" event={"ID":"cafd1877-5d84-4f0a-aa25-a26378cd2732","Type":"ContainerStarted","Data":"d0fe18eb233250ade83c5b0da785dec4e4de3ef158e38d99baa094420fa22381"}
Nov 03 01:20:35.300887 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:35.300863    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cnz49"
Nov 03 01:20:35.402328 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:35.402300    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cafd1877-5d84-4f0a-aa25-a26378cd2732-kube-api-access-6p9md\" (UniqueName: \"kubernetes.io/projected/cafd1877-5d84-4f0a-aa25-a26378cd2732-kube-api-access-6p9md\") pod \"cafd1877-5d84-4f0a-aa25-a26378cd2732\" (UID: \"cafd1877-5d84-4f0a-aa25-a26378cd2732\") "
Nov 03 01:20:35.402478 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:35.402348    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cafd1877-5d84-4f0a-aa25-a26378cd2732-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cafd1877-5d84-4f0a-aa25-a26378cd2732-maas-api-xks-ca\") pod \"cafd1877-5d84-4f0a-aa25-a26378cd2732\" (UID: \"cafd1877-5d84-4f0a-aa25-a26378cd2732\") "
Nov 03 01:20:35.402478 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:35.402463    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cafd1877-5d84-4f0a-aa25-a26378cd2732-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cafd1877-5d84-4f0a-aa25-a26378cd2732-maas-api-openshift-ca\") pod \"cafd1877-5d84-4f0a-aa25-a26378cd2732\" (UID: \"cafd1877-5d84-4f0a-aa25-a26378cd2732\") "
Nov 03 01:20:35.402799 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:35.402771    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cafd1877-5d84-4f0a-aa25-a26378cd2732-maas-api-openshift-ca" pod "cafd1877-5d84-4f0a-aa25-a26378cd2732" (UID: "cafd1877-5d84-4f0a-aa25-a26378cd2732"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:20:35.402904 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:35.402859    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cafd1877-5d84-4f0a-aa25-a26378cd2732-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:20:35.404428 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:35.404405    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cafd1877-5d84-4f0a-aa25-a26378cd2732-maas-api-xks-ca" pod "cafd1877-5d84-4f0a-aa25-a26378cd2732" (UID: "cafd1877-5d84-4f0a-aa25-a26378cd2732"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:20:35.404538 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:35.404441    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafd1877-5d84-4f0a-aa25-a26378cd2732-kube-api-access-6p9md" pod "cafd1877-5d84-4f0a-aa25-a26378cd2732" (UID: "cafd1877-5d84-4f0a-aa25-a26378cd2732"). InnerVolumeSpecName "kube-api-access-6p9md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:20:35.503924 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:35.503891    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p9md\" (UniqueName: \"kubernetes.io/projected/cafd1877-5d84-4f0a-aa25-a26378cd2732-kube-api-access-6p9md\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:20:35.504028 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:35.503929    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cafd1877-5d84-4f0a-aa25-a26378cd2732-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:20:36.167383 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:36.167354    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cnz49"
Nov 03 01:20:36.167383 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:36.167375    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cnz49" event={"ID":"cafd1877-5d84-4f0a-aa25-a26378cd2732","Type":"ContainerDied","Data":"d0fe18eb233250ade83c5b0da785dec4e4de3ef158e38d99baa094420fa22381"}
Nov 03 01:20:36.167589 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:20:36.167417    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0fe18eb233250ade83c5b0da785dec4e4de3ef158e38d99baa094420fa22381"
Nov 03 01:20:37.363857 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:20:37.363816    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:21:07.362625 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:21:07.362531    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:21:37.364675 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:21:37.364637    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:21:37.692148 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:37.692053    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1479c112-9g6p5"]
Nov 03 01:21:37.692573 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:37.692552    5012 cpu_manager.go:414] "containerMap: removing container" podUID="cafd1877-5d84-4f0a-aa25-a26378cd2732" containerName="revoke-keys"
Nov 03 01:21:37.692573 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:37.692572    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cafd1877-5d84-4f0a-aa25-a26378cd2732" containerName="revoke-keys"
Nov 03 01:21:37.692737 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:37.692643    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="cafd1877-5d84-4f0a-aa25-a26378cd2732" containerName="revoke-keys"
Nov 03 01:21:37.696053 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:37.696028    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1479c112-9g6p5"
Nov 03 01:21:37.710111 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:37.710068    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lsxz\" (UniqueName: \"kubernetes.io/projected/7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca-kube-api-access-8lsxz\") pod \"maas-api-revoke-sub-1479c112-9g6p5\" (UID: \"7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1479c112-9g6p5"
Nov 03 01:21:37.710303 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:37.710172    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1479c112-9g6p5\" (UID: \"7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1479c112-9g6p5"
Nov 03 01:21:37.710303 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:37.710284    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1479c112-9g6p5\" (UID: \"7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1479c112-9g6p5"
Nov 03 01:21:37.711743 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:37.711719    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:21:37.722452 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:37.722412    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:21:37.722651 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:37.722540    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:21:37.723017 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:37.722989    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:21:37.726796 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:37.726295    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1479c112-9g6p5"]
Nov 03 01:21:37.811459 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:37.811401    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1479c112-9g6p5\" (UID: \"7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1479c112-9g6p5"
Nov 03 01:21:37.811691 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:37.811477    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1479c112-9g6p5\" (UID: \"7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1479c112-9g6p5"
Nov 03 01:21:37.811691 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:37.811624    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lsxz\" (UniqueName: \"kubernetes.io/projected/7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca-kube-api-access-8lsxz\") pod \"maas-api-revoke-sub-1479c112-9g6p5\" (UID: \"7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1479c112-9g6p5"
Nov 03 01:21:37.812264 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:37.812240    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1479c112-9g6p5\" (UID: \"7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1479c112-9g6p5"
Nov 03 01:21:37.813919 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:37.813889    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1479c112-9g6p5\" (UID: \"7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1479c112-9g6p5"
Nov 03 01:21:37.831199 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:37.831165    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lsxz\" (UniqueName: \"kubernetes.io/projected/7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca-kube-api-access-8lsxz\") pod \"maas-api-revoke-sub-1479c112-9g6p5\" (UID: \"7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1479c112-9g6p5"
Nov 03 01:21:38.010106 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:38.010010    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1479c112-9g6p5"
Nov 03 01:21:38.313201 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:38.313159    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1479c112-9g6p5"]
Nov 03 01:21:38.315306 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:21:38.315271    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea06228_f4d0_4a50_ab6d_74a5f8dd0cca.slice/crio-580de28ec8b627b0a0ac0c6318c46ca321fb3ba21124bccadefc11d6efbcf3ae WatchSource:0}: Error finding container 580de28ec8b627b0a0ac0c6318c46ca321fb3ba21124bccadefc11d6efbcf3ae: Status 404 returned error can't find the container with id 580de28ec8b627b0a0ac0c6318c46ca321fb3ba21124bccadefc11d6efbcf3ae
Nov 03 01:21:38.616165 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:38.616120    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1479c112-9g6p5" event={"ID":"7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca","Type":"ContainerStarted","Data":"580de28ec8b627b0a0ac0c6318c46ca321fb3ba21124bccadefc11d6efbcf3ae"}
Nov 03 01:21:39.624592 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:39.624554    5012 generic.go:356] "Generic (PLEG): container finished" podID="7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca" containerID="7fd2af65d7bae9fa7aea3e8014dde362b0874e3449cf4fe999f33605a3837541" exitCode=0
Nov 03 01:21:39.625103 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:39.624637    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1479c112-9g6p5" event={"ID":"7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca","Type":"ContainerDied","Data":"7fd2af65d7bae9fa7aea3e8014dde362b0874e3449cf4fe999f33605a3837541"}
Nov 03 01:21:40.774117 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:40.774090    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1479c112-9g6p5"
Nov 03 01:21:40.835294 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:40.835264    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca-kube-api-access-8lsxz\" (UniqueName: \"kubernetes.io/projected/7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca-kube-api-access-8lsxz\") pod \"7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca\" (UID: \"7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca\") "
Nov 03 01:21:40.835490 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:40.835319    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca-maas-api-openshift-ca\") pod \"7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca\" (UID: \"7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca\") "
Nov 03 01:21:40.835490 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:40.835389    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca-maas-api-xks-ca\") pod \"7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca\" (UID: \"7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca\") "
Nov 03 01:21:40.835772 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:40.835742    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca-maas-api-openshift-ca" pod "7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca" (UID: "7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:21:40.837550 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:40.837521    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca-maas-api-xks-ca" pod "7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca" (UID: "7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:21:40.837681 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:40.837550    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca-kube-api-access-8lsxz" pod "7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca" (UID: "7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca"). InnerVolumeSpecName "kube-api-access-8lsxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:21:40.936599 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:40.936530    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:21:40.936599 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:40.936559    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lsxz\" (UniqueName: \"kubernetes.io/projected/7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca-kube-api-access-8lsxz\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:21:40.936599 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:40.936569    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:21:41.638490 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:41.638459    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1479c112-9g6p5"
Nov 03 01:21:41.638673 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:41.638494    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1479c112-9g6p5" event={"ID":"7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca","Type":"ContainerDied","Data":"580de28ec8b627b0a0ac0c6318c46ca321fb3ba21124bccadefc11d6efbcf3ae"}
Nov 03 01:21:41.638673 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:41.638543    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="580de28ec8b627b0a0ac0c6318c46ca321fb3ba21124bccadefc11d6efbcf3ae"
Nov 03 01:21:52.346931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:52.346441    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-243c406d-m6ls4"]
Nov 03 01:21:52.347432 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:52.347088    5012 cpu_manager.go:414] "containerMap: removing container" podUID="7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca" containerName="revoke-keys"
Nov 03 01:21:52.347432 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:52.347107    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca" containerName="revoke-keys"
Nov 03 01:21:52.347432 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:52.347249    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca" containerName="revoke-keys"
Nov 03 01:21:52.365514 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:52.365487    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-243c406d-m6ls4"
Nov 03 01:21:52.371930 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:52.366926    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-243c406d-m6ls4"]
Nov 03 01:21:52.409930 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:52.394148    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:21:52.409930 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:52.394198    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:21:52.409930 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:52.394244    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:21:52.409930 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:52.394310    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:21:52.454942 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:52.446135    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn4w4\" (UniqueName: \"kubernetes.io/projected/5f359a86-ebbb-4474-bb81-eea347c7ecb5-kube-api-access-bn4w4\") pod \"maas-api-revoke-sub-243c406d-m6ls4\" (UID: \"5f359a86-ebbb-4474-bb81-eea347c7ecb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-243c406d-m6ls4"
Nov 03 01:21:52.456932 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:52.455269    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f359a86-ebbb-4474-bb81-eea347c7ecb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-243c406d-m6ls4\" (UID: \"5f359a86-ebbb-4474-bb81-eea347c7ecb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-243c406d-m6ls4"
Nov 03 01:21:52.457184 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:52.457148    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f359a86-ebbb-4474-bb81-eea347c7ecb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-243c406d-m6ls4\" (UID: \"5f359a86-ebbb-4474-bb81-eea347c7ecb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-243c406d-m6ls4"
Nov 03 01:21:52.558528 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:52.558490    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn4w4\" (UniqueName: \"kubernetes.io/projected/5f359a86-ebbb-4474-bb81-eea347c7ecb5-kube-api-access-bn4w4\") pod \"maas-api-revoke-sub-243c406d-m6ls4\" (UID: \"5f359a86-ebbb-4474-bb81-eea347c7ecb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-243c406d-m6ls4"
Nov 03 01:21:52.558816 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:52.558790    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f359a86-ebbb-4474-bb81-eea347c7ecb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-243c406d-m6ls4\" (UID: \"5f359a86-ebbb-4474-bb81-eea347c7ecb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-243c406d-m6ls4"
Nov 03 01:21:52.559102 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:52.559066    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f359a86-ebbb-4474-bb81-eea347c7ecb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-243c406d-m6ls4\" (UID: \"5f359a86-ebbb-4474-bb81-eea347c7ecb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-243c406d-m6ls4"
Nov 03 01:21:52.560168 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:52.560144    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f359a86-ebbb-4474-bb81-eea347c7ecb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-243c406d-m6ls4\" (UID: \"5f359a86-ebbb-4474-bb81-eea347c7ecb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-243c406d-m6ls4"
Nov 03 01:21:52.564949 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:52.562076    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f359a86-ebbb-4474-bb81-eea347c7ecb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-243c406d-m6ls4\" (UID: \"5f359a86-ebbb-4474-bb81-eea347c7ecb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-243c406d-m6ls4"
Nov 03 01:21:52.576014 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:52.575989    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn4w4\" (UniqueName: \"kubernetes.io/projected/5f359a86-ebbb-4474-bb81-eea347c7ecb5-kube-api-access-bn4w4\") pod \"maas-api-revoke-sub-243c406d-m6ls4\" (UID: \"5f359a86-ebbb-4474-bb81-eea347c7ecb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-243c406d-m6ls4"
Nov 03 01:21:52.684464 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:52.684379    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-243c406d-m6ls4"
Nov 03 01:21:53.153603 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:53.153563    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-243c406d-m6ls4"]
Nov 03 01:21:53.596941 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:53.593062    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hstf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:21:53.596941 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:53.593160    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4" podUID="c2788e33-32b9-47f6-8d84-d8663aa07a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:21:53.765769 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:53.765689    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-243c406d-m6ls4" event={"ID":"5f359a86-ebbb-4474-bb81-eea347c7ecb5","Type":"ContainerStarted","Data":"304c61641c671bb2e00f5789cf2285aae02db1c415b0f08d06cc285d940b3dab"}
Nov 03 01:21:54.773705 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:54.773669    5012 generic.go:356] "Generic (PLEG): container finished" podID="5f359a86-ebbb-4474-bb81-eea347c7ecb5" containerID="7acafd912b59d41dbba3350b1dc1b859f88ce9850f26a91f4e9675fe5c1b68fa" exitCode=0
Nov 03 01:21:54.774124 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:54.773753    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-243c406d-m6ls4" event={"ID":"5f359a86-ebbb-4474-bb81-eea347c7ecb5","Type":"ContainerDied","Data":"7acafd912b59d41dbba3350b1dc1b859f88ce9850f26a91f4e9675fe5c1b68fa"}
Nov 03 01:21:56.231590 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:56.231258    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-243c406d-m6ls4"
Nov 03 01:21:56.372752 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:56.372715    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f359a86-ebbb-4474-bb81-eea347c7ecb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f359a86-ebbb-4474-bb81-eea347c7ecb5-maas-api-openshift-ca\") pod \"5f359a86-ebbb-4474-bb81-eea347c7ecb5\" (UID: \"5f359a86-ebbb-4474-bb81-eea347c7ecb5\") "
Nov 03 01:21:56.373093 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:56.373066    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f359a86-ebbb-4474-bb81-eea347c7ecb5-kube-api-access-bn4w4\" (UniqueName: \"kubernetes.io/projected/5f359a86-ebbb-4474-bb81-eea347c7ecb5-kube-api-access-bn4w4\") pod \"5f359a86-ebbb-4474-bb81-eea347c7ecb5\" (UID: \"5f359a86-ebbb-4474-bb81-eea347c7ecb5\") "
Nov 03 01:21:56.373372 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:56.373349    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f359a86-ebbb-4474-bb81-eea347c7ecb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f359a86-ebbb-4474-bb81-eea347c7ecb5-maas-api-xks-ca\") pod \"5f359a86-ebbb-4474-bb81-eea347c7ecb5\" (UID: \"5f359a86-ebbb-4474-bb81-eea347c7ecb5\") "
Nov 03 01:21:56.374545 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:56.374515    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f359a86-ebbb-4474-bb81-eea347c7ecb5-maas-api-openshift-ca" pod "5f359a86-ebbb-4474-bb81-eea347c7ecb5" (UID: "5f359a86-ebbb-4474-bb81-eea347c7ecb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:21:56.376799 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:56.376771    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f359a86-ebbb-4474-bb81-eea347c7ecb5-kube-api-access-bn4w4" pod "5f359a86-ebbb-4474-bb81-eea347c7ecb5" (UID: "5f359a86-ebbb-4474-bb81-eea347c7ecb5"). InnerVolumeSpecName "kube-api-access-bn4w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:21:56.376937 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:56.376771    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f359a86-ebbb-4474-bb81-eea347c7ecb5-maas-api-xks-ca" pod "5f359a86-ebbb-4474-bb81-eea347c7ecb5" (UID: "5f359a86-ebbb-4474-bb81-eea347c7ecb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:21:56.474648 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:56.474586    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f359a86-ebbb-4474-bb81-eea347c7ecb5-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:21:56.474648 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:56.474609    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn4w4\" (UniqueName: \"kubernetes.io/projected/5f359a86-ebbb-4474-bb81-eea347c7ecb5-kube-api-access-bn4w4\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:21:56.474648 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:56.474620    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f359a86-ebbb-4474-bb81-eea347c7ecb5-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:21:56.841982 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:56.828352    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-243c406d-m6ls4" event={"ID":"5f359a86-ebbb-4474-bb81-eea347c7ecb5","Type":"ContainerDied","Data":"304c61641c671bb2e00f5789cf2285aae02db1c415b0f08d06cc285d940b3dab"}
Nov 03 01:21:56.841982 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:56.828415    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="304c61641c671bb2e00f5789cf2285aae02db1c415b0f08d06cc285d940b3dab"
Nov 03 01:21:56.841982 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:21:56.828486    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-243c406d-m6ls4"
Nov 03 01:22:02.028141 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:02.028109    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c02b89e9-4kwrs"]
Nov 03 01:22:02.038655 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:02.038628    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c02b89e9-4kwrs"]
Nov 03 01:22:03.242797 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:03.242759    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69e14a92-1722-49b6-b26a-f45d08375476" path="/var/lib/kubelet/pods/69e14a92-1722-49b6-b26a-f45d08375476/volumes"
Nov 03 01:22:04.649843 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.649803    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"]
Nov 03 01:22:04.651150 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.651119    5012 cpu_manager.go:414] "containerMap: removing container" podUID="5f359a86-ebbb-4474-bb81-eea347c7ecb5" containerName="revoke-keys"
Nov 03 01:22:04.651259 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.651151    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f359a86-ebbb-4474-bb81-eea347c7ecb5" containerName="revoke-keys"
Nov 03 01:22:04.651303 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.651260    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f359a86-ebbb-4474-bb81-eea347c7ecb5" containerName="revoke-keys"
Nov 03 01:22:04.655204 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.655184    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.668037 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.668019    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-b9d7ac-openshift-default-dockercfg-mjssb\""
Nov 03 01:22:04.681944 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.681890    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"]
Nov 03 01:22:04.747681 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.747653    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-workload-certs\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.747811 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.747699    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-data\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.747811 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.747717    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-workload-socket\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.747811 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.747761    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-token\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.747964 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.747876    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlmz7\" (UniqueName: \"kubernetes.io/projected/a551781b-7dda-421e-92a0-f0fcc49d5cca-kube-api-access-jlmz7\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.747964 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.747933    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-envoy\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.747964 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.747955    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-credential-socket\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.748121 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.747993    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-podinfo\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.748121 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.748046    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a551781b-7dda-421e-92a0-f0fcc49d5cca-istiod-ca-cert\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.849102 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.849064    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlmz7\" (UniqueName: \"kubernetes.io/projected/a551781b-7dda-421e-92a0-f0fcc49d5cca-kube-api-access-jlmz7\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.849299 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.849120    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-envoy\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.849299 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.849149    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-credential-socket\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.849299 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.849189    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-podinfo\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.849299 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.849218    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a551781b-7dda-421e-92a0-f0fcc49d5cca-istiod-ca-cert\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.849604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.849339    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-workload-certs\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.849604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.849407    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-data\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.849604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.849486    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-workload-socket\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.849604 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.849557    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-token\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.849842 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.849710    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-data\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.850020 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.849991    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-workload-socket\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.850153 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.850135    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a551781b-7dda-421e-92a0-f0fcc49d5cca-istiod-ca-cert\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.850269 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.850232    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-credential-socket\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.850386 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.850253    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-workload-certs\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.851691 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.851673    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-envoy\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.852211 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.852189    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-podinfo\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.869682 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.869647    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlmz7\" (UniqueName: \"kubernetes.io/projected/a551781b-7dda-421e-92a0-f0fcc49d5cca-kube-api-access-jlmz7\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.869786 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.869772    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-token\") pod \"e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") " pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:04.969199 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:04.969108    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:05.284657 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:05.284630    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"]
Nov 03 01:22:05.286154 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:22:05.286129    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda551781b_7dda_421e_92a0_f0fcc49d5cca.slice/crio-39711e05e4ba77dedfb0ae76c44127e9522314eaa789cf50b7fe534aa88f82b1 WatchSource:0}: Error finding container 39711e05e4ba77dedfb0ae76c44127e9522314eaa789cf50b7fe534aa88f82b1: Status 404 returned error can't find the container with id 39711e05e4ba77dedfb0ae76c44127e9522314eaa789cf50b7fe534aa88f82b1
Nov 03 01:22:05.288092 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:05.288055    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:22:05.288181 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:05.288122    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:22:05.288181 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:05.288156    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:22:05.885048 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:05.885014    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb" event={"ID":"a551781b-7dda-421e-92a0-f0fcc49d5cca","Type":"ContainerStarted","Data":"929c1734b4e3f39572ca4617cc4f89e5e6f024ac8a871e0c937b18d83ae15815"}
Nov 03 01:22:05.885455 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:05.885076    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb" event={"ID":"a551781b-7dda-421e-92a0-f0fcc49d5cca","Type":"ContainerStarted","Data":"39711e05e4ba77dedfb0ae76c44127e9522314eaa789cf50b7fe534aa88f82b1"}
Nov 03 01:22:05.935148 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:05.935098    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb" podStartSLOduration=1.935085039 podStartE2EDuration="1.935085039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:22:05.932798207 +0000 UTC m=+4608.812382250" watchObservedRunningTime="2026-11-03 01:22:05.935085039 +0000 UTC m=+4608.814669082"
Nov 03 01:22:05.970273 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:05.970248    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:06.975013 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:06.974984    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:07.362369 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:22:07.362338    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:22:07.897531 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:07.897499    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:07.898364 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:07.898344    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:09.991496 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:09.991462    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g"]
Nov 03 01:22:09.995749 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:09.995727    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g"
Nov 03 01:22:10.016081 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:10.016054    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g"]
Nov 03 01:22:10.022158 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:10.022130    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:22:10.022290 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:10.022232    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-b9d7ac-serving-cert\""
Nov 03 01:22:10.022520 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:10.022496    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:22:10.023177 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:10.023146    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-x7pjw\""
Nov 03 01:22:10.023177 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:10.023164    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 01:22:10.100340 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:10.100310    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ab0e67d-9055-42f4-8e42-62b61cc05a50-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g\" (UID: \"0ab0e67d-9055-42f4-8e42-62b61cc05a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g"
Nov 03 01:22:10.100463 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:10.100364    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab0e67d-9055-42f4-8e42-62b61cc05a50-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g\" (UID: \"0ab0e67d-9055-42f4-8e42-62b61cc05a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g"
Nov 03 01:22:10.100463 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:10.100456    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rzzl\" (UniqueName: \"kubernetes.io/projected/0ab0e67d-9055-42f4-8e42-62b61cc05a50-kube-api-access-9rzzl\") pod \"maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g\" (UID: \"0ab0e67d-9055-42f4-8e42-62b61cc05a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g"
Nov 03 01:22:10.100552 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:10.100482    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ab0e67d-9055-42f4-8e42-62b61cc05a50-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g\" (UID: \"0ab0e67d-9055-42f4-8e42-62b61cc05a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g"
Nov 03 01:22:10.201107 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:10.201077    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ab0e67d-9055-42f4-8e42-62b61cc05a50-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g\" (UID: \"0ab0e67d-9055-42f4-8e42-62b61cc05a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g"
Nov 03 01:22:10.201272 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:10.201129    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab0e67d-9055-42f4-8e42-62b61cc05a50-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g\" (UID: \"0ab0e67d-9055-42f4-8e42-62b61cc05a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g"
Nov 03 01:22:10.201272 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:10.201180    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rzzl\" (UniqueName: \"kubernetes.io/projected/0ab0e67d-9055-42f4-8e42-62b61cc05a50-kube-api-access-9rzzl\") pod \"maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g\" (UID: \"0ab0e67d-9055-42f4-8e42-62b61cc05a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g"
Nov 03 01:22:10.201272 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:10.201208    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ab0e67d-9055-42f4-8e42-62b61cc05a50-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g\" (UID: \"0ab0e67d-9055-42f4-8e42-62b61cc05a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g"
Nov 03 01:22:10.201885 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:10.201846    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab0e67d-9055-42f4-8e42-62b61cc05a50-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g\" (UID: \"0ab0e67d-9055-42f4-8e42-62b61cc05a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g"
Nov 03 01:22:10.204102 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:10.204070    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ab0e67d-9055-42f4-8e42-62b61cc05a50-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g\" (UID: \"0ab0e67d-9055-42f4-8e42-62b61cc05a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g"
Nov 03 01:22:10.204314 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:10.204290    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ab0e67d-9055-42f4-8e42-62b61cc05a50-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g\" (UID: \"0ab0e67d-9055-42f4-8e42-62b61cc05a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g"
Nov 03 01:22:10.222267 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:10.222240    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rzzl\" (UniqueName: \"kubernetes.io/projected/0ab0e67d-9055-42f4-8e42-62b61cc05a50-kube-api-access-9rzzl\") pod \"maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g\" (UID: \"0ab0e67d-9055-42f4-8e42-62b61cc05a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g"
Nov 03 01:22:10.309622 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:10.309554    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g"
Nov 03 01:22:10.614151 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:10.614123    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g"]
Nov 03 01:22:10.615722 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:22:10.615696    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab0e67d_9055_42f4_8e42_62b61cc05a50.slice/crio-1fec6c58b7997cf73b3abfc0773a4bbc29a3f95435ef221bfed3288335d90f13 WatchSource:0}: Error finding container 1fec6c58b7997cf73b3abfc0773a4bbc29a3f95435ef221bfed3288335d90f13: Status 404 returned error can't find the container with id 1fec6c58b7997cf73b3abfc0773a4bbc29a3f95435ef221bfed3288335d90f13
Nov 03 01:22:10.616938 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:10.616894    5012 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 01:22:10.920066 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:10.919982    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g" event={"ID":"0ab0e67d-9055-42f4-8e42-62b61cc05a50","Type":"ContainerStarted","Data":"1fec6c58b7997cf73b3abfc0773a4bbc29a3f95435ef221bfed3288335d90f13"}
Nov 03 01:22:11.928275 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:11.928236    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g" event={"ID":"0ab0e67d-9055-42f4-8e42-62b61cc05a50","Type":"ContainerStarted","Data":"b8e78d3132473cfc9a72962ecc9ef5312450aac7affc36f73527d5c7112d5de3"}
Nov 03 01:22:11.928751 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:11.928306    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g"
Nov 03 01:22:11.966566 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:11.966513    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g" podStartSLOduration=2.598986081 podStartE2EDuration="2.966499745s" totalImagesPullingTime="367.513664ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:22:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:22:11.963874819 +0000 UTC m=+4614.843458859" watchObservedRunningTime="2026-11-03 01:22:11.966499745 +0000 UTC m=+4614.846083785"
Nov 03 01:22:14.074242 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:14.074201    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b46b34e6-h2988"]
Nov 03 01:22:14.108931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:14.108716    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b46b34e6-h2988"]
Nov 03 01:22:15.242720 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:15.242670    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2" path="/var/lib/kubelet/pods/bb7b9c5a-6bd5-4921-b7f7-44fb44b4bec2/volumes"
Nov 03 01:22:17.941249 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:17.941217    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g"
Nov 03 01:22:22.898935 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:22.898845    5012 patch_prober.go:29] interesting pod/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Nov 03 01:22:22.898935 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:22.898901    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb" podUID="a551781b-7dda-421e-92a0-f0fcc49d5cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Nov 03 01:22:23.028606 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:23.028573    5012 generic.go:356] "Generic (PLEG): container finished" podID="a551781b-7dda-421e-92a0-f0fcc49d5cca" containerID="929c1734b4e3f39572ca4617cc4f89e5e6f024ac8a871e0c937b18d83ae15815" exitCode=0
Nov 03 01:22:23.028792 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:23.028646    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb" event={"ID":"a551781b-7dda-421e-92a0-f0fcc49d5cca","Type":"ContainerDied","Data":"929c1734b4e3f39572ca4617cc4f89e5e6f024ac8a871e0c937b18d83ae15815"}
Nov 03 01:22:23.029104 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:23.029087    5012 scope.go:122] "RemoveContainer" containerID="929c1734b4e3f39572ca4617cc4f89e5e6f024ac8a871e0c937b18d83ae15815"
Nov 03 01:22:23.030297 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:23.030267    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:22:23.030390 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:23.030350    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:22:23.030472 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:23.030425    5012 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:22:24.038177 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:24.038130    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb" event={"ID":"a551781b-7dda-421e-92a0-f0fcc49d5cca","Type":"ContainerStarted","Data":"36d998b5bd9314465d47b1e9f0adaff94ce40a5a421ad8ef047a88780e1d8640"}
Nov 03 01:22:24.969592 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:24.969556    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:24.969849 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:24.969820    5012 patch_prober.go:29] interesting pod/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Nov 03 01:22:24.969946 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:24.969893    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb" podUID="a551781b-7dda-421e-92a0-f0fcc49d5cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Nov 03 01:22:25.969834 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:25.969796    5012 patch_prober.go:29] interesting pod/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Nov 03 01:22:25.970288 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:25.969848    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb" podUID="a551781b-7dda-421e-92a0-f0fcc49d5cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Nov 03 01:22:26.970421 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:26.970377    5012 patch_prober.go:29] interesting pod/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Nov 03 01:22:26.970850 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:26.970438    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb" podUID="a551781b-7dda-421e-92a0-f0fcc49d5cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Nov 03 01:22:27.970554 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:27.970465    5012 patch_prober.go:29] interesting pod/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Nov 03 01:22:27.970554 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:27.970532    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb" podUID="a551781b-7dda-421e-92a0-f0fcc49d5cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Nov 03 01:22:28.970528 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:28.970493    5012 patch_prober.go:29] interesting pod/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Nov 03 01:22:28.970708 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:28.970548    5012 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb" podUID="a551781b-7dda-421e-92a0-f0fcc49d5cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Nov 03 01:22:29.974149 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:29.974117    5012 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:29.974652 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:29.974480    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:29.975137 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:22:29.975120    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:22:37.362571 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:22:37.362530    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:23:07.362233 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:23:07.362188    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:23:15.444443 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:15.444404    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g"]
Nov 03 01:23:15.445016 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:15.444588    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g" podUID="0ab0e67d-9055-42f4-8e42-62b61cc05a50" containerName="maas-api" containerID="cri-o://b8e78d3132473cfc9a72962ecc9ef5312450aac7affc36f73527d5c7112d5de3" gracePeriod=30
Nov 03 01:23:15.933230 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:15.933202    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g"
Nov 03 01:23:15.950817 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:15.950794    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ab0e67d-9055-42f4-8e42-62b61cc05a50-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ab0e67d-9055-42f4-8e42-62b61cc05a50-maas-api-tls\") pod \"0ab0e67d-9055-42f4-8e42-62b61cc05a50\" (UID: \"0ab0e67d-9055-42f4-8e42-62b61cc05a50\") "
Nov 03 01:23:15.950973 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:15.950833    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ab0e67d-9055-42f4-8e42-62b61cc05a50-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ab0e67d-9055-42f4-8e42-62b61cc05a50-metrics-certs\") pod \"0ab0e67d-9055-42f4-8e42-62b61cc05a50\" (UID: \"0ab0e67d-9055-42f4-8e42-62b61cc05a50\") "
Nov 03 01:23:15.950973 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:15.950939    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab0e67d-9055-42f4-8e42-62b61cc05a50-kube-api-access-9rzzl\" (UniqueName: \"kubernetes.io/projected/0ab0e67d-9055-42f4-8e42-62b61cc05a50-kube-api-access-9rzzl\") pod \"0ab0e67d-9055-42f4-8e42-62b61cc05a50\" (UID: \"0ab0e67d-9055-42f4-8e42-62b61cc05a50\") "
Nov 03 01:23:15.951112 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:15.950977    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ab0e67d-9055-42f4-8e42-62b61cc05a50-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab0e67d-9055-42f4-8e42-62b61cc05a50-openshift-service-ca\") pod \"0ab0e67d-9055-42f4-8e42-62b61cc05a50\" (UID: \"0ab0e67d-9055-42f4-8e42-62b61cc05a50\") "
Nov 03 01:23:15.951410 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:15.951384    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab0e67d-9055-42f4-8e42-62b61cc05a50-openshift-service-ca" pod "0ab0e67d-9055-42f4-8e42-62b61cc05a50" (UID: "0ab0e67d-9055-42f4-8e42-62b61cc05a50"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:23:15.952959 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:15.952932    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ab0e67d-9055-42f4-8e42-62b61cc05a50-maas-api-tls" pod "0ab0e67d-9055-42f4-8e42-62b61cc05a50" (UID: "0ab0e67d-9055-42f4-8e42-62b61cc05a50"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:23:15.953064 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:15.953001    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ab0e67d-9055-42f4-8e42-62b61cc05a50-metrics-certs" pod "0ab0e67d-9055-42f4-8e42-62b61cc05a50" (UID: "0ab0e67d-9055-42f4-8e42-62b61cc05a50"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:23:15.953064 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:15.953009    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab0e67d-9055-42f4-8e42-62b61cc05a50-kube-api-access-9rzzl" pod "0ab0e67d-9055-42f4-8e42-62b61cc05a50" (UID: "0ab0e67d-9055-42f4-8e42-62b61cc05a50"). InnerVolumeSpecName "kube-api-access-9rzzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:23:16.052285 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:16.052228    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rzzl\" (UniqueName: \"kubernetes.io/projected/0ab0e67d-9055-42f4-8e42-62b61cc05a50-kube-api-access-9rzzl\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:23:16.052285 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:16.052251    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab0e67d-9055-42f4-8e42-62b61cc05a50-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:23:16.052285 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:16.052262    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ab0e67d-9055-42f4-8e42-62b61cc05a50-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:23:16.052285 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:16.052271    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ab0e67d-9055-42f4-8e42-62b61cc05a50-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:23:16.368712 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:16.368682    5012 generic.go:356] "Generic (PLEG): container finished" podID="0ab0e67d-9055-42f4-8e42-62b61cc05a50" containerID="b8e78d3132473cfc9a72962ecc9ef5312450aac7affc36f73527d5c7112d5de3" exitCode=0
Nov 03 01:23:16.368891 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:16.368726    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g"
Nov 03 01:23:16.368891 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:16.368730    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g" event={"ID":"0ab0e67d-9055-42f4-8e42-62b61cc05a50","Type":"ContainerDied","Data":"b8e78d3132473cfc9a72962ecc9ef5312450aac7affc36f73527d5c7112d5de3"}
Nov 03 01:23:16.368891 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:16.368847    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g" event={"ID":"0ab0e67d-9055-42f4-8e42-62b61cc05a50","Type":"ContainerDied","Data":"1fec6c58b7997cf73b3abfc0773a4bbc29a3f95435ef221bfed3288335d90f13"}
Nov 03 01:23:16.368891 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:16.368869    5012 scope.go:122] "RemoveContainer" containerID="b8e78d3132473cfc9a72962ecc9ef5312450aac7affc36f73527d5c7112d5de3"
Nov 03 01:23:16.379859 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:16.379836    5012 scope.go:122] "RemoveContainer" containerID="b8e78d3132473cfc9a72962ecc9ef5312450aac7affc36f73527d5c7112d5de3"
Nov 03 01:23:16.380139 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:23:16.380120    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8e78d3132473cfc9a72962ecc9ef5312450aac7affc36f73527d5c7112d5de3\": container with ID starting with b8e78d3132473cfc9a72962ecc9ef5312450aac7affc36f73527d5c7112d5de3 not found: ID does not exist" containerID="b8e78d3132473cfc9a72962ecc9ef5312450aac7affc36f73527d5c7112d5de3"
Nov 03 01:23:16.380191 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:16.380148    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8e78d3132473cfc9a72962ecc9ef5312450aac7affc36f73527d5c7112d5de3"} err="failed to get container status \"b8e78d3132473cfc9a72962ecc9ef5312450aac7affc36f73527d5c7112d5de3\": rpc error: code = NotFound desc = could not find container \"b8e78d3132473cfc9a72962ecc9ef5312450aac7affc36f73527d5c7112d5de3\": container with ID starting with b8e78d3132473cfc9a72962ecc9ef5312450aac7affc36f73527d5c7112d5de3 not found: ID does not exist"
Nov 03 01:23:16.416058 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:16.416032    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g"]
Nov 03 01:23:16.426214 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:16.426190    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-9885g"]
Nov 03 01:23:17.242468 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:17.242433    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ab0e67d-9055-42f4-8e42-62b61cc05a50" path="/var/lib/kubelet/pods/0ab0e67d-9055-42f4-8e42-62b61cc05a50/volumes"
Nov 03 01:23:21.936333 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:21.936296    5012 scope.go:122] "RemoveContainer" containerID="072cc46a2b589c884ffa1f369ee6f78951cf6b0bc575bfa36bbd1fdff10f34b1"
Nov 03 01:23:21.945792 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:23:21.945773    5012 scope.go:122] "RemoveContainer" containerID="eb446078f097a6d9058990610dfd04b86f6c8cfce50a51cf251d01160b86d2b7"
Nov 03 01:23:37.362462 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:23:37.362425    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:24:04.032232 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:04.032150    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4hpff"]
Nov 03 01:24:04.042306 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:04.042277    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4hpff"]
Nov 03 01:24:05.242852 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:05.242813    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cf27a36-9d3f-421a-9950-6aec1e21af96" path="/var/lib/kubelet/pods/4cf27a36-9d3f-421a-9950-6aec1e21af96/volumes"
Nov 03 01:24:07.362714 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:24:07.362678    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:24:08.913852 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:08.913762    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"]
Nov 03 01:24:08.914362 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:08.914040    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb" podUID="a551781b-7dda-421e-92a0-f0fcc49d5cca" containerName="istio-proxy" containerID="cri-o://36d998b5bd9314465d47b1e9f0adaff94ce40a5a421ad8ef047a88780e1d8640" gracePeriod=30
Nov 03 01:24:14.206865 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.206833    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:24:14.273497 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.273471    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-workload-socket\") pod \"a551781b-7dda-421e-92a0-f0fcc49d5cca\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") "
Nov 03 01:24:14.273656 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.273525    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a551781b-7dda-421e-92a0-f0fcc49d5cca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a551781b-7dda-421e-92a0-f0fcc49d5cca-istiod-ca-cert\") pod \"a551781b-7dda-421e-92a0-f0fcc49d5cca\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") "
Nov 03 01:24:14.273656 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.273554    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-token\" (UniqueName: \"kubernetes.io/projected/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-token\") pod \"a551781b-7dda-421e-92a0-f0fcc49d5cca\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") "
Nov 03 01:24:14.273656 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.273595    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-data\") pod \"a551781b-7dda-421e-92a0-f0fcc49d5cca\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") "
Nov 03 01:24:14.273656 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.273635    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-envoy\") pod \"a551781b-7dda-421e-92a0-f0fcc49d5cca\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") "
Nov 03 01:24:14.273871 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.273675    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-credential-socket\") pod \"a551781b-7dda-421e-92a0-f0fcc49d5cca\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") "
Nov 03 01:24:14.273871 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.273743    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-podinfo\") pod \"a551781b-7dda-421e-92a0-f0fcc49d5cca\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") "
Nov 03 01:24:14.273871 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.273773    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-workload-socket" pod "a551781b-7dda-421e-92a0-f0fcc49d5cca" (UID: "a551781b-7dda-421e-92a0-f0fcc49d5cca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:24:14.273871 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.273796    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-workload-certs\") pod \"a551781b-7dda-421e-92a0-f0fcc49d5cca\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") "
Nov 03 01:24:14.273871 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.273853    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a551781b-7dda-421e-92a0-f0fcc49d5cca-kube-api-access-jlmz7\" (UniqueName: \"kubernetes.io/projected/a551781b-7dda-421e-92a0-f0fcc49d5cca-kube-api-access-jlmz7\") pod \"a551781b-7dda-421e-92a0-f0fcc49d5cca\" (UID: \"a551781b-7dda-421e-92a0-f0fcc49d5cca\") "
Nov 03 01:24:14.274181 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.273952    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-data" pod "a551781b-7dda-421e-92a0-f0fcc49d5cca" (UID: "a551781b-7dda-421e-92a0-f0fcc49d5cca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:24:14.274181 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.274085    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a551781b-7dda-421e-92a0-f0fcc49d5cca-istiod-ca-cert" pod "a551781b-7dda-421e-92a0-f0fcc49d5cca" (UID: "a551781b-7dda-421e-92a0-f0fcc49d5cca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:24:14.274329 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.274215    5012 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-workload-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:24:14.274329 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.274235    5012 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a551781b-7dda-421e-92a0-f0fcc49d5cca-istiod-ca-cert\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:24:14.274509 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.274367    5012 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-data\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:24:14.274509 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.274375    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-workload-certs" pod "a551781b-7dda-421e-92a0-f0fcc49d5cca" (UID: "a551781b-7dda-421e-92a0-f0fcc49d5cca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:24:14.274654 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.274627    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-credential-socket" pod "a551781b-7dda-421e-92a0-f0fcc49d5cca" (UID: "a551781b-7dda-421e-92a0-f0fcc49d5cca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:24:14.275977 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.275952    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-token" pod "a551781b-7dda-421e-92a0-f0fcc49d5cca" (UID: "a551781b-7dda-421e-92a0-f0fcc49d5cca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:24:14.276410 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.276383    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-envoy" pod "a551781b-7dda-421e-92a0-f0fcc49d5cca" (UID: "a551781b-7dda-421e-92a0-f0fcc49d5cca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:24:14.276516 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.276481    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a551781b-7dda-421e-92a0-f0fcc49d5cca-kube-api-access-jlmz7" pod "a551781b-7dda-421e-92a0-f0fcc49d5cca" (UID: "a551781b-7dda-421e-92a0-f0fcc49d5cca"). InnerVolumeSpecName "kube-api-access-jlmz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:24:14.276579 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.276558    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-podinfo" pod "a551781b-7dda-421e-92a0-f0fcc49d5cca" (UID: "a551781b-7dda-421e-92a0-f0fcc49d5cca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:24:14.374852 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.374832    5012 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-envoy\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:24:14.374852 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.374853    5012 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-credential-socket\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:24:14.375020 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.374863    5012 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-podinfo\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:24:14.375020 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.374872    5012 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a551781b-7dda-421e-92a0-f0fcc49d5cca-workload-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:24:14.375020 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.374880    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlmz7\" (UniqueName: \"kubernetes.io/projected/a551781b-7dda-421e-92a0-f0fcc49d5cca-kube-api-access-jlmz7\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:24:14.375020 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.374891    5012 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a551781b-7dda-421e-92a0-f0fcc49d5cca-istio-token\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:24:14.756651 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.756616    5012 generic.go:356] "Generic (PLEG): container finished" podID="a551781b-7dda-421e-92a0-f0fcc49d5cca" containerID="36d998b5bd9314465d47b1e9f0adaff94ce40a5a421ad8ef047a88780e1d8640" exitCode=0
Nov 03 01:24:14.756833 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.756664    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"
Nov 03 01:24:14.756833 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.756701    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb" event={"ID":"a551781b-7dda-421e-92a0-f0fcc49d5cca","Type":"ContainerDied","Data":"36d998b5bd9314465d47b1e9f0adaff94ce40a5a421ad8ef047a88780e1d8640"}
Nov 03 01:24:14.756833 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.756758    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb" event={"ID":"a551781b-7dda-421e-92a0-f0fcc49d5cca","Type":"ContainerDied","Data":"39711e05e4ba77dedfb0ae76c44127e9522314eaa789cf50b7fe534aa88f82b1"}
Nov 03 01:24:14.756833 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.756783    5012 scope.go:122] "RemoveContainer" containerID="36d998b5bd9314465d47b1e9f0adaff94ce40a5a421ad8ef047a88780e1d8640"
Nov 03 01:24:14.767179 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.767162    5012 scope.go:122] "RemoveContainer" containerID="929c1734b4e3f39572ca4617cc4f89e5e6f024ac8a871e0c937b18d83ae15815"
Nov 03 01:24:14.775766 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.775748    5012 scope.go:122] "RemoveContainer" containerID="36d998b5bd9314465d47b1e9f0adaff94ce40a5a421ad8ef047a88780e1d8640"
Nov 03 01:24:14.776054 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:24:14.776032    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d998b5bd9314465d47b1e9f0adaff94ce40a5a421ad8ef047a88780e1d8640\": container with ID starting with 36d998b5bd9314465d47b1e9f0adaff94ce40a5a421ad8ef047a88780e1d8640 not found: ID does not exist" containerID="36d998b5bd9314465d47b1e9f0adaff94ce40a5a421ad8ef047a88780e1d8640"
Nov 03 01:24:14.776119 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.776061    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d998b5bd9314465d47b1e9f0adaff94ce40a5a421ad8ef047a88780e1d8640"} err="failed to get container status \"36d998b5bd9314465d47b1e9f0adaff94ce40a5a421ad8ef047a88780e1d8640\": rpc error: code = NotFound desc = could not find container \"36d998b5bd9314465d47b1e9f0adaff94ce40a5a421ad8ef047a88780e1d8640\": container with ID starting with 36d998b5bd9314465d47b1e9f0adaff94ce40a5a421ad8ef047a88780e1d8640 not found: ID does not exist"
Nov 03 01:24:14.776119 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.776080    5012 scope.go:122] "RemoveContainer" containerID="929c1734b4e3f39572ca4617cc4f89e5e6f024ac8a871e0c937b18d83ae15815"
Nov 03 01:24:14.776328 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:24:14.776309    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"929c1734b4e3f39572ca4617cc4f89e5e6f024ac8a871e0c937b18d83ae15815\": container with ID starting with 929c1734b4e3f39572ca4617cc4f89e5e6f024ac8a871e0c937b18d83ae15815 not found: ID does not exist" containerID="929c1734b4e3f39572ca4617cc4f89e5e6f024ac8a871e0c937b18d83ae15815"
Nov 03 01:24:14.776386 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.776337    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"929c1734b4e3f39572ca4617cc4f89e5e6f024ac8a871e0c937b18d83ae15815"} err="failed to get container status \"929c1734b4e3f39572ca4617cc4f89e5e6f024ac8a871e0c937b18d83ae15815\": rpc error: code = NotFound desc = could not find container \"929c1734b4e3f39572ca4617cc4f89e5e6f024ac8a871e0c937b18d83ae15815\": container with ID starting with 929c1734b4e3f39572ca4617cc4f89e5e6f024ac8a871e0c937b18d83ae15815 not found: ID does not exist"
Nov 03 01:24:14.814006 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.813979    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"]
Nov 03 01:24:14.824037 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:14.824015    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b9d7ac-openshift-default-57778fb5b59l4cb"]
Nov 03 01:24:15.243777 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:15.243745    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a551781b-7dda-421e-92a0-f0fcc49d5cca" path="/var/lib/kubelet/pods/a551781b-7dda-421e-92a0-f0fcc49d5cca/volumes"
Nov 03 01:24:19.649795 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:19.649754    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-cdvd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:24:19.650456 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:19.650418    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9" podUID="a125db3f-e711-4e19-ac5d-cbd3827be603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:24:37.362400 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:24:37.362362    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:24:44.024052 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:44.024018    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xgg58"]
Nov 03 01:24:44.034216 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:44.034181    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xgg58"]
Nov 03 01:24:45.241923 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:45.241877    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b78873b-8271-4452-9b32-f761dd4d7285" path="/var/lib/kubelet/pods/5b78873b-8271-4452-9b32-f761dd4d7285/volumes"
Nov 03 01:24:49.245127 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:49.245090    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b9n22"]
Nov 03 01:24:49.245713 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:49.245691    5012 cpu_manager.go:414] "containerMap: removing container" podUID="a551781b-7dda-421e-92a0-f0fcc49d5cca" containerName="istio-proxy"
Nov 03 01:24:49.245764 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:49.245717    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a551781b-7dda-421e-92a0-f0fcc49d5cca" containerName="istio-proxy"
Nov 03 01:24:49.245801 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:49.245781    5012 cpu_manager.go:414] "containerMap: removing container" podUID="0ab0e67d-9055-42f4-8e42-62b61cc05a50" containerName="maas-api"
Nov 03 01:24:49.245801 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:49.245790    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab0e67d-9055-42f4-8e42-62b61cc05a50" containerName="maas-api"
Nov 03 01:24:49.245940 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:49.245901    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="a551781b-7dda-421e-92a0-f0fcc49d5cca" containerName="istio-proxy"
Nov 03 01:24:49.245987 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:49.245947    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab0e67d-9055-42f4-8e42-62b61cc05a50" containerName="maas-api"
Nov 03 01:24:49.245987 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:49.245959    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="a551781b-7dda-421e-92a0-f0fcc49d5cca" containerName="istio-proxy"
Nov 03 01:24:49.249883 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:49.249861    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b9n22"
Nov 03 01:24:49.258113 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:49.258046    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b9n22"]
Nov 03 01:24:49.271277 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:49.264247    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:24:49.271277 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:49.264325    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:24:49.271277 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:49.264431    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:24:49.273272 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:49.273250    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:24:49.287270 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:49.287239    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bce404fd-5ded-4ffe-b555-459d81edab3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-b9n22\" (UID: \"bce404fd-5ded-4ffe-b555-459d81edab3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b9n22"
Nov 03 01:24:49.287372 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:49.287331    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bce404fd-5ded-4ffe-b555-459d81edab3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-b9n22\" (UID: \"bce404fd-5ded-4ffe-b555-459d81edab3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b9n22"
Nov 03 01:24:49.287415 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:49.287387    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5q22\" (UniqueName: \"kubernetes.io/projected/bce404fd-5ded-4ffe-b555-459d81edab3c-kube-api-access-q5q22\") pod \"maas-api-revoke-sub-452b6d7b-b9n22\" (UID: \"bce404fd-5ded-4ffe-b555-459d81edab3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b9n22"
Nov 03 01:24:49.387952 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:49.387897    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bce404fd-5ded-4ffe-b555-459d81edab3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-b9n22\" (UID: \"bce404fd-5ded-4ffe-b555-459d81edab3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b9n22"
Nov 03 01:24:49.388156 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:49.387985    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bce404fd-5ded-4ffe-b555-459d81edab3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-b9n22\" (UID: \"bce404fd-5ded-4ffe-b555-459d81edab3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b9n22"
Nov 03 01:24:49.388156 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:49.388022    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5q22\" (UniqueName: \"kubernetes.io/projected/bce404fd-5ded-4ffe-b555-459d81edab3c-kube-api-access-q5q22\") pod \"maas-api-revoke-sub-452b6d7b-b9n22\" (UID: \"bce404fd-5ded-4ffe-b555-459d81edab3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b9n22"
Nov 03 01:24:49.388918 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:49.388866    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bce404fd-5ded-4ffe-b555-459d81edab3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-b9n22\" (UID: \"bce404fd-5ded-4ffe-b555-459d81edab3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b9n22"
Nov 03 01:24:49.390546 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:49.390514    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bce404fd-5ded-4ffe-b555-459d81edab3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-b9n22\" (UID: \"bce404fd-5ded-4ffe-b555-459d81edab3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b9n22"
Nov 03 01:24:49.406352 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:49.406322    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5q22\" (UniqueName: \"kubernetes.io/projected/bce404fd-5ded-4ffe-b555-459d81edab3c-kube-api-access-q5q22\") pod \"maas-api-revoke-sub-452b6d7b-b9n22\" (UID: \"bce404fd-5ded-4ffe-b555-459d81edab3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b9n22"
Nov 03 01:24:49.572528 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:49.572432    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b9n22"
Nov 03 01:24:49.892130 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:49.892093    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b9n22"]
Nov 03 01:24:49.897694 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:24:49.897659    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce404fd_5ded_4ffe_b555_459d81edab3c.slice/crio-24ef5f7bd5654147ddf1f4a9cd6299dd024287ce334cbf9d69c4966879cd8849 WatchSource:0}: Error finding container 24ef5f7bd5654147ddf1f4a9cd6299dd024287ce334cbf9d69c4966879cd8849: Status 404 returned error can't find the container with id 24ef5f7bd5654147ddf1f4a9cd6299dd024287ce334cbf9d69c4966879cd8849
Nov 03 01:24:50.072934 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:50.072175    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b9n22" event={"ID":"bce404fd-5ded-4ffe-b555-459d81edab3c","Type":"ContainerStarted","Data":"24ef5f7bd5654147ddf1f4a9cd6299dd024287ce334cbf9d69c4966879cd8849"}
Nov 03 01:24:51.083665 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:51.083618    5012 generic.go:356] "Generic (PLEG): container finished" podID="bce404fd-5ded-4ffe-b555-459d81edab3c" containerID="86bda7a1a3db942ef9c4a6514c940cb531e17c8dad7f0e0a4743fd0c59d42c28" exitCode=0
Nov 03 01:24:51.084131 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:51.083750    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b9n22" event={"ID":"bce404fd-5ded-4ffe-b555-459d81edab3c","Type":"ContainerDied","Data":"86bda7a1a3db942ef9c4a6514c940cb531e17c8dad7f0e0a4743fd0c59d42c28"}
Nov 03 01:24:52.300578 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:52.300552    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b9n22"
Nov 03 01:24:52.372563 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:52.372492    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bce404fd-5ded-4ffe-b555-459d81edab3c-kube-api-access-q5q22\" (UniqueName: \"kubernetes.io/projected/bce404fd-5ded-4ffe-b555-459d81edab3c-kube-api-access-q5q22\") pod \"bce404fd-5ded-4ffe-b555-459d81edab3c\" (UID: \"bce404fd-5ded-4ffe-b555-459d81edab3c\") "
Nov 03 01:24:52.372713 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:52.372610    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bce404fd-5ded-4ffe-b555-459d81edab3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bce404fd-5ded-4ffe-b555-459d81edab3c-maas-api-xks-ca\") pod \"bce404fd-5ded-4ffe-b555-459d81edab3c\" (UID: \"bce404fd-5ded-4ffe-b555-459d81edab3c\") "
Nov 03 01:24:52.372713 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:52.372660    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bce404fd-5ded-4ffe-b555-459d81edab3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bce404fd-5ded-4ffe-b555-459d81edab3c-maas-api-openshift-ca\") pod \"bce404fd-5ded-4ffe-b555-459d81edab3c\" (UID: \"bce404fd-5ded-4ffe-b555-459d81edab3c\") "
Nov 03 01:24:52.373083 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:52.373058    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bce404fd-5ded-4ffe-b555-459d81edab3c-maas-api-openshift-ca" pod "bce404fd-5ded-4ffe-b555-459d81edab3c" (UID: "bce404fd-5ded-4ffe-b555-459d81edab3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:24:52.374695 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:52.374673    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce404fd-5ded-4ffe-b555-459d81edab3c-kube-api-access-q5q22" pod "bce404fd-5ded-4ffe-b555-459d81edab3c" (UID: "bce404fd-5ded-4ffe-b555-459d81edab3c"). InnerVolumeSpecName "kube-api-access-q5q22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:24:52.374812 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:52.374741    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bce404fd-5ded-4ffe-b555-459d81edab3c-maas-api-xks-ca" pod "bce404fd-5ded-4ffe-b555-459d81edab3c" (UID: "bce404fd-5ded-4ffe-b555-459d81edab3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:24:52.473793 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:52.473759    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bce404fd-5ded-4ffe-b555-459d81edab3c-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:24:52.473793 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:52.473788    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bce404fd-5ded-4ffe-b555-459d81edab3c-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:24:52.473793 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:52.473799    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5q22\" (UniqueName: \"kubernetes.io/projected/bce404fd-5ded-4ffe-b555-459d81edab3c-kube-api-access-q5q22\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:24:53.110979 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:53.110946    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b9n22"
Nov 03 01:24:53.110979 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:53.110964    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b9n22" event={"ID":"bce404fd-5ded-4ffe-b555-459d81edab3c","Type":"ContainerDied","Data":"24ef5f7bd5654147ddf1f4a9cd6299dd024287ce334cbf9d69c4966879cd8849"}
Nov 03 01:24:53.111254 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:24:53.111023    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24ef5f7bd5654147ddf1f4a9cd6299dd024287ce334cbf9d69c4966879cd8849"
Nov 03 01:25:06.287554 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:06.287516    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rhttb"]
Nov 03 01:25:06.288235 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:06.288209    5012 cpu_manager.go:414] "containerMap: removing container" podUID="bce404fd-5ded-4ffe-b555-459d81edab3c" containerName="revoke-keys"
Nov 03 01:25:06.288312 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:06.288237    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bce404fd-5ded-4ffe-b555-459d81edab3c" containerName="revoke-keys"
Nov 03 01:25:06.288312 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:06.288271    5012 cpu_manager.go:414] "containerMap: removing container" podUID="a551781b-7dda-421e-92a0-f0fcc49d5cca" containerName="istio-proxy"
Nov 03 01:25:06.288312 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:06.288279    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a551781b-7dda-421e-92a0-f0fcc49d5cca" containerName="istio-proxy"
Nov 03 01:25:06.288479 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:06.288379    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="bce404fd-5ded-4ffe-b555-459d81edab3c" containerName="revoke-keys"
Nov 03 01:25:06.291553 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:06.291527    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rhttb"
Nov 03 01:25:06.315466 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:06.315443    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:25:06.315740 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:06.315712    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:25:06.315854 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:06.315821    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:25:06.316000 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:06.315869    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:25:06.320791 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:06.320762    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rhttb"]
Nov 03 01:25:06.418081 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:06.418053    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cszc8\" (UniqueName: \"kubernetes.io/projected/d69a1ce9-cd85-4f0d-b6a7-5317dc21c645-kube-api-access-cszc8\") pod \"maas-api-revoke-sub-3e31a506-rhttb\" (UID: \"d69a1ce9-cd85-4f0d-b6a7-5317dc21c645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rhttb"
Nov 03 01:25:06.418231 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:06.418109    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69a1ce9-cd85-4f0d-b6a7-5317dc21c645-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-rhttb\" (UID: \"d69a1ce9-cd85-4f0d-b6a7-5317dc21c645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rhttb"
Nov 03 01:25:06.418231 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:06.418210    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69a1ce9-cd85-4f0d-b6a7-5317dc21c645-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-rhttb\" (UID: \"d69a1ce9-cd85-4f0d-b6a7-5317dc21c645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rhttb"
Nov 03 01:25:06.519357 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:06.519321    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69a1ce9-cd85-4f0d-b6a7-5317dc21c645-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-rhttb\" (UID: \"d69a1ce9-cd85-4f0d-b6a7-5317dc21c645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rhttb"
Nov 03 01:25:06.519519 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:06.519396    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cszc8\" (UniqueName: \"kubernetes.io/projected/d69a1ce9-cd85-4f0d-b6a7-5317dc21c645-kube-api-access-cszc8\") pod \"maas-api-revoke-sub-3e31a506-rhttb\" (UID: \"d69a1ce9-cd85-4f0d-b6a7-5317dc21c645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rhttb"
Nov 03 01:25:06.519519 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:06.519465    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69a1ce9-cd85-4f0d-b6a7-5317dc21c645-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-rhttb\" (UID: \"d69a1ce9-cd85-4f0d-b6a7-5317dc21c645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rhttb"
Nov 03 01:25:06.520031 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:06.520006    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69a1ce9-cd85-4f0d-b6a7-5317dc21c645-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-rhttb\" (UID: \"d69a1ce9-cd85-4f0d-b6a7-5317dc21c645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rhttb"
Nov 03 01:25:06.527841 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:06.521777    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69a1ce9-cd85-4f0d-b6a7-5317dc21c645-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-rhttb\" (UID: \"d69a1ce9-cd85-4f0d-b6a7-5317dc21c645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rhttb"
Nov 03 01:25:06.536744 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:06.536713    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cszc8\" (UniqueName: \"kubernetes.io/projected/d69a1ce9-cd85-4f0d-b6a7-5317dc21c645-kube-api-access-cszc8\") pod \"maas-api-revoke-sub-3e31a506-rhttb\" (UID: \"d69a1ce9-cd85-4f0d-b6a7-5317dc21c645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rhttb"
Nov 03 01:25:06.604463 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:06.604442    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rhttb"
Nov 03 01:25:06.901625 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:06.901597    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rhttb"]
Nov 03 01:25:06.904613 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:25:06.904572    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd69a1ce9_cd85_4f0d_b6a7_5317dc21c645.slice/crio-9b17674494c2b2f2abdca88d5eae0c0c2ccdf1f249032711c6f77311405a1625 WatchSource:0}: Error finding container 9b17674494c2b2f2abdca88d5eae0c0c2ccdf1f249032711c6f77311405a1625: Status 404 returned error can't find the container with id 9b17674494c2b2f2abdca88d5eae0c0c2ccdf1f249032711c6f77311405a1625
Nov 03 01:25:07.202168 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:07.202123    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rhttb" event={"ID":"d69a1ce9-cd85-4f0d-b6a7-5317dc21c645","Type":"ContainerStarted","Data":"9b17674494c2b2f2abdca88d5eae0c0c2ccdf1f249032711c6f77311405a1625"}
Nov 03 01:25:07.369614 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:25:07.369567    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:25:08.234948 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:08.231109    5012 generic.go:356] "Generic (PLEG): container finished" podID="d69a1ce9-cd85-4f0d-b6a7-5317dc21c645" containerID="bc485aca5adcfa30658e8275931416b280890add7759648f7d8a96b72a1355a3" exitCode=0
Nov 03 01:25:08.234948 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:08.231313    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rhttb" event={"ID":"d69a1ce9-cd85-4f0d-b6a7-5317dc21c645","Type":"ContainerDied","Data":"bc485aca5adcfa30658e8275931416b280890add7759648f7d8a96b72a1355a3"}
Nov 03 01:25:09.037267 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:09.037172    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2zzwf"]
Nov 03 01:25:09.049764 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:09.049731    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2zzwf"]
Nov 03 01:25:09.260053 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:09.260021    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faaf71e2-e4b4-4763-8899-c21ded229b62" path="/var/lib/kubelet/pods/faaf71e2-e4b4-4763-8899-c21ded229b62/volumes"
Nov 03 01:25:09.475375 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:09.475350    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rhttb"
Nov 03 01:25:09.570799 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:09.570767    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d69a1ce9-cd85-4f0d-b6a7-5317dc21c645-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69a1ce9-cd85-4f0d-b6a7-5317dc21c645-maas-api-openshift-ca\") pod \"d69a1ce9-cd85-4f0d-b6a7-5317dc21c645\" (UID: \"d69a1ce9-cd85-4f0d-b6a7-5317dc21c645\") "
Nov 03 01:25:09.570945 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:09.570808    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d69a1ce9-cd85-4f0d-b6a7-5317dc21c645-kube-api-access-cszc8\" (UniqueName: \"kubernetes.io/projected/d69a1ce9-cd85-4f0d-b6a7-5317dc21c645-kube-api-access-cszc8\") pod \"d69a1ce9-cd85-4f0d-b6a7-5317dc21c645\" (UID: \"d69a1ce9-cd85-4f0d-b6a7-5317dc21c645\") "
Nov 03 01:25:09.570945 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:09.570858    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d69a1ce9-cd85-4f0d-b6a7-5317dc21c645-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69a1ce9-cd85-4f0d-b6a7-5317dc21c645-maas-api-xks-ca\") pod \"d69a1ce9-cd85-4f0d-b6a7-5317dc21c645\" (UID: \"d69a1ce9-cd85-4f0d-b6a7-5317dc21c645\") "
Nov 03 01:25:09.571216 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:09.571185    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d69a1ce9-cd85-4f0d-b6a7-5317dc21c645-maas-api-openshift-ca" pod "d69a1ce9-cd85-4f0d-b6a7-5317dc21c645" (UID: "d69a1ce9-cd85-4f0d-b6a7-5317dc21c645"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:25:09.571333 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:09.571304    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69a1ce9-cd85-4f0d-b6a7-5317dc21c645-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:25:09.572999 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:09.572974    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d69a1ce9-cd85-4f0d-b6a7-5317dc21c645-maas-api-xks-ca" pod "d69a1ce9-cd85-4f0d-b6a7-5317dc21c645" (UID: "d69a1ce9-cd85-4f0d-b6a7-5317dc21c645"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:25:09.573088 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:09.573014    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d69a1ce9-cd85-4f0d-b6a7-5317dc21c645-kube-api-access-cszc8" pod "d69a1ce9-cd85-4f0d-b6a7-5317dc21c645" (UID: "d69a1ce9-cd85-4f0d-b6a7-5317dc21c645"). InnerVolumeSpecName "kube-api-access-cszc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:25:09.672779 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:09.672711    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cszc8\" (UniqueName: \"kubernetes.io/projected/d69a1ce9-cd85-4f0d-b6a7-5317dc21c645-kube-api-access-cszc8\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:25:09.672779 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:09.672743    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69a1ce9-cd85-4f0d-b6a7-5317dc21c645-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:25:10.262626 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:10.262591    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rhttb"
Nov 03 01:25:10.263061 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:10.262594    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rhttb" event={"ID":"d69a1ce9-cd85-4f0d-b6a7-5317dc21c645","Type":"ContainerDied","Data":"9b17674494c2b2f2abdca88d5eae0c0c2ccdf1f249032711c6f77311405a1625"}
Nov 03 01:25:10.263061 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:10.262719    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b17674494c2b2f2abdca88d5eae0c0c2ccdf1f249032711c6f77311405a1625"
Nov 03 01:25:18.825759 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:18.825726    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 01:25:18.851672 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:18.851646    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 01:25:18.863122 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:18.863103    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 01:25:18.871446 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:18.871423    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 01:25:19.699052 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:19.698996    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-cdvd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:25:19.699368 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:19.699317    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9" podUID="a125db3f-e711-4e19-ac5d-cbd3827be603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:25:21.991576 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:21.991540    5012 scope.go:122] "RemoveContainer" containerID="cff21a24a8c7dea9946bdc5c1167814d6515e045959f3a838bec6c7f4359d116"
Nov 03 01:25:22.001418 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:22.001400    5012 scope.go:122] "RemoveContainer" containerID="a8e8800c4774f296ff8527bf14f95409739925ad3778a497161b657745473802"
Nov 03 01:25:25.056992 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:25.056953    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x6v55"]
Nov 03 01:25:25.062931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:25.060279    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-x6v55"]
Nov 03 01:25:25.254180 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:25.254135    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f34a890-46ff-4ab9-a5be-b7ba879cc7fa" path="/var/lib/kubelet/pods/5f34a890-46ff-4ab9-a5be-b7ba879cc7fa/volumes"
Nov 03 01:25:26.296421 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:26.296385    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ef9843-rtcwf"]
Nov 03 01:25:26.297621 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:26.297588    5012 cpu_manager.go:414] "containerMap: removing container" podUID="d69a1ce9-cd85-4f0d-b6a7-5317dc21c645" containerName="revoke-keys"
Nov 03 01:25:26.297791 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:26.297776    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d69a1ce9-cd85-4f0d-b6a7-5317dc21c645" containerName="revoke-keys"
Nov 03 01:25:26.298062 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:26.298046    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="d69a1ce9-cd85-4f0d-b6a7-5317dc21c645" containerName="revoke-keys"
Nov 03 01:25:26.304263 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:26.304234    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ef9843-rtcwf"
Nov 03 01:25:26.318181 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:26.318152    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ef9843-rtcwf"]
Nov 03 01:25:26.324889 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:26.324865    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:25:26.325096 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:26.324938    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:25:26.325096 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:26.324983    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:25:26.327661 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:26.327639    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:25:26.379738 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:26.379714    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bff2a9d-336f-43b2-ba62-14d7bca26a9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48ef9843-rtcwf\" (UID: \"0bff2a9d-336f-43b2-ba62-14d7bca26a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ef9843-rtcwf"
Nov 03 01:25:26.379853 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:26.379785    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bff2a9d-336f-43b2-ba62-14d7bca26a9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48ef9843-rtcwf\" (UID: \"0bff2a9d-336f-43b2-ba62-14d7bca26a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ef9843-rtcwf"
Nov 03 01:25:26.379939 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:26.379880    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxjqd\" (UniqueName: \"kubernetes.io/projected/0bff2a9d-336f-43b2-ba62-14d7bca26a9a-kube-api-access-nxjqd\") pod \"maas-api-revoke-sub-48ef9843-rtcwf\" (UID: \"0bff2a9d-336f-43b2-ba62-14d7bca26a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ef9843-rtcwf"
Nov 03 01:25:26.480714 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:26.480690    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxjqd\" (UniqueName: \"kubernetes.io/projected/0bff2a9d-336f-43b2-ba62-14d7bca26a9a-kube-api-access-nxjqd\") pod \"maas-api-revoke-sub-48ef9843-rtcwf\" (UID: \"0bff2a9d-336f-43b2-ba62-14d7bca26a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ef9843-rtcwf"
Nov 03 01:25:26.480840 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:26.480783    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bff2a9d-336f-43b2-ba62-14d7bca26a9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48ef9843-rtcwf\" (UID: \"0bff2a9d-336f-43b2-ba62-14d7bca26a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ef9843-rtcwf"
Nov 03 01:25:26.480940 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:26.480894    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bff2a9d-336f-43b2-ba62-14d7bca26a9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48ef9843-rtcwf\" (UID: \"0bff2a9d-336f-43b2-ba62-14d7bca26a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ef9843-rtcwf"
Nov 03 01:25:26.481530 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:26.481510    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bff2a9d-336f-43b2-ba62-14d7bca26a9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48ef9843-rtcwf\" (UID: \"0bff2a9d-336f-43b2-ba62-14d7bca26a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ef9843-rtcwf"
Nov 03 01:25:26.483136 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:26.483117    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bff2a9d-336f-43b2-ba62-14d7bca26a9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48ef9843-rtcwf\" (UID: \"0bff2a9d-336f-43b2-ba62-14d7bca26a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ef9843-rtcwf"
Nov 03 01:25:26.497985 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:26.497962    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxjqd\" (UniqueName: \"kubernetes.io/projected/0bff2a9d-336f-43b2-ba62-14d7bca26a9a-kube-api-access-nxjqd\") pod \"maas-api-revoke-sub-48ef9843-rtcwf\" (UID: \"0bff2a9d-336f-43b2-ba62-14d7bca26a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ef9843-rtcwf"
Nov 03 01:25:26.616849 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:26.616826    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ef9843-rtcwf"
Nov 03 01:25:26.912958 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:26.912930    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ef9843-rtcwf"]
Nov 03 01:25:26.914103 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:25:26.914076    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bff2a9d_336f_43b2_ba62_14d7bca26a9a.slice/crio-385adad68cad1422071a7bbd7efd6d0ea515cb5e988cd452af7ff20c481a7088 WatchSource:0}: Error finding container 385adad68cad1422071a7bbd7efd6d0ea515cb5e988cd452af7ff20c481a7088: Status 404 returned error can't find the container with id 385adad68cad1422071a7bbd7efd6d0ea515cb5e988cd452af7ff20c481a7088
Nov 03 01:25:27.440249 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:27.440210    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ef9843-rtcwf" event={"ID":"0bff2a9d-336f-43b2-ba62-14d7bca26a9a","Type":"ContainerStarted","Data":"a31ffea9d36469b3e3ac2d90ec9c653862513695bc49153ce70bc88f598151a0"}
Nov 03 01:25:27.440628 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:27.440270    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ef9843-rtcwf" event={"ID":"0bff2a9d-336f-43b2-ba62-14d7bca26a9a","Type":"ContainerStarted","Data":"385adad68cad1422071a7bbd7efd6d0ea515cb5e988cd452af7ff20c481a7088"}
Nov 03 01:25:27.485116 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:27.485064    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ef9843-rtcwf" podStartSLOduration=1.4850527279999999 podStartE2EDuration="1.485052728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:25:27.473808134 +0000 UTC m=+4810.353392177" watchObservedRunningTime="2026-11-03 01:25:27.485052728 +0000 UTC m=+4810.364636770"
Nov 03 01:25:28.447981 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:28.447947    5012 generic.go:356] "Generic (PLEG): container finished" podID="0bff2a9d-336f-43b2-ba62-14d7bca26a9a" containerID="a31ffea9d36469b3e3ac2d90ec9c653862513695bc49153ce70bc88f598151a0" exitCode=0
Nov 03 01:25:28.448385 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:28.448042    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ef9843-rtcwf" event={"ID":"0bff2a9d-336f-43b2-ba62-14d7bca26a9a","Type":"ContainerDied","Data":"a31ffea9d36469b3e3ac2d90ec9c653862513695bc49153ce70bc88f598151a0"}
Nov 03 01:25:29.587257 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:29.587222    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ef9843-rtcwf"
Nov 03 01:25:29.711016 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:29.710940    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bff2a9d-336f-43b2-ba62-14d7bca26a9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bff2a9d-336f-43b2-ba62-14d7bca26a9a-maas-api-openshift-ca\") pod \"0bff2a9d-336f-43b2-ba62-14d7bca26a9a\" (UID: \"0bff2a9d-336f-43b2-ba62-14d7bca26a9a\") "
Nov 03 01:25:29.711158 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:29.711020    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bff2a9d-336f-43b2-ba62-14d7bca26a9a-kube-api-access-nxjqd\" (UniqueName: \"kubernetes.io/projected/0bff2a9d-336f-43b2-ba62-14d7bca26a9a-kube-api-access-nxjqd\") pod \"0bff2a9d-336f-43b2-ba62-14d7bca26a9a\" (UID: \"0bff2a9d-336f-43b2-ba62-14d7bca26a9a\") "
Nov 03 01:25:29.711158 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:29.711116    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bff2a9d-336f-43b2-ba62-14d7bca26a9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bff2a9d-336f-43b2-ba62-14d7bca26a9a-maas-api-xks-ca\") pod \"0bff2a9d-336f-43b2-ba62-14d7bca26a9a\" (UID: \"0bff2a9d-336f-43b2-ba62-14d7bca26a9a\") "
Nov 03 01:25:29.711329 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:29.711305    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bff2a9d-336f-43b2-ba62-14d7bca26a9a-maas-api-openshift-ca" pod "0bff2a9d-336f-43b2-ba62-14d7bca26a9a" (UID: "0bff2a9d-336f-43b2-ba62-14d7bca26a9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:25:29.713249 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:29.713223    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bff2a9d-336f-43b2-ba62-14d7bca26a9a-maas-api-xks-ca" pod "0bff2a9d-336f-43b2-ba62-14d7bca26a9a" (UID: "0bff2a9d-336f-43b2-ba62-14d7bca26a9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:25:29.713362 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:29.713268    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bff2a9d-336f-43b2-ba62-14d7bca26a9a-kube-api-access-nxjqd" pod "0bff2a9d-336f-43b2-ba62-14d7bca26a9a" (UID: "0bff2a9d-336f-43b2-ba62-14d7bca26a9a"). InnerVolumeSpecName "kube-api-access-nxjqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:25:29.812624 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:29.812595    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxjqd\" (UniqueName: \"kubernetes.io/projected/0bff2a9d-336f-43b2-ba62-14d7bca26a9a-kube-api-access-nxjqd\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:25:29.812624 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:29.812621    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bff2a9d-336f-43b2-ba62-14d7bca26a9a-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:25:29.812771 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:29.812631    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bff2a9d-336f-43b2-ba62-14d7bca26a9a-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:25:30.461717 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:30.461674    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ef9843-rtcwf" event={"ID":"0bff2a9d-336f-43b2-ba62-14d7bca26a9a","Type":"ContainerDied","Data":"385adad68cad1422071a7bbd7efd6d0ea515cb5e988cd452af7ff20c481a7088"}
Nov 03 01:25:30.461969 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:30.461735    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="385adad68cad1422071a7bbd7efd6d0ea515cb5e988cd452af7ff20c481a7088"
Nov 03 01:25:30.461969 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:30.461694    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ef9843-rtcwf"
Nov 03 01:25:36.032413 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:36.032380    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cnz49"]
Nov 03 01:25:36.042432 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:36.042403    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cnz49"]
Nov 03 01:25:37.242880 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:25:37.242842    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cafd1877-5d84-4f0a-aa25-a26378cd2732" path="/var/lib/kubelet/pods/cafd1877-5d84-4f0a-aa25-a26378cd2732/volumes"
Nov 03 01:25:37.362746 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:25:37.362708    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:26:07.362816 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:26:07.362778    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:26:08.238402 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:08.238338    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-885tl"]
Nov 03 01:26:13.239589 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:13.239553    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-885tl" podUID="df410080-3e08-4f8d-b1d3-84db851d8782" containerName="payload-processing" containerID="cri-o://51ce4bc91216d66f795a38a60af249fef168c1bffc3b3b670aad38674e602941" gracePeriod=25
Nov 03 01:26:13.482943 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:13.482920    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-885tl"
Nov 03 01:26:13.583123 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:13.583039    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df410080-3e08-4f8d-b1d3-84db851d8782-kube-api-access-8l87z\" (UniqueName: \"kubernetes.io/projected/df410080-3e08-4f8d-b1d3-84db851d8782-kube-api-access-8l87z\") pod \"df410080-3e08-4f8d-b1d3-84db851d8782\" (UID: \"df410080-3e08-4f8d-b1d3-84db851d8782\") "
Nov 03 01:26:13.583280 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:13.583199    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df410080-3e08-4f8d-b1d3-84db851d8782-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df410080-3e08-4f8d-b1d3-84db851d8782-plugins-config-volume\") pod \"df410080-3e08-4f8d-b1d3-84db851d8782\" (UID: \"df410080-3e08-4f8d-b1d3-84db851d8782\") "
Nov 03 01:26:13.583592 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:13.583567    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df410080-3e08-4f8d-b1d3-84db851d8782-plugins-config-volume" pod "df410080-3e08-4f8d-b1d3-84db851d8782" (UID: "df410080-3e08-4f8d-b1d3-84db851d8782"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:26:13.585093 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:13.585074    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df410080-3e08-4f8d-b1d3-84db851d8782-kube-api-access-8l87z" pod "df410080-3e08-4f8d-b1d3-84db851d8782" (UID: "df410080-3e08-4f8d-b1d3-84db851d8782"). InnerVolumeSpecName "kube-api-access-8l87z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:26:13.684677 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:13.684636    5012 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df410080-3e08-4f8d-b1d3-84db851d8782-plugins-config-volume\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:26:13.684677 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:13.684677    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l87z\" (UniqueName: \"kubernetes.io/projected/df410080-3e08-4f8d-b1d3-84db851d8782-kube-api-access-8l87z\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:26:13.725113 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:13.725082    5012 generic.go:356] "Generic (PLEG): container finished" podID="df410080-3e08-4f8d-b1d3-84db851d8782" containerID="51ce4bc91216d66f795a38a60af249fef168c1bffc3b3b670aad38674e602941" exitCode=0
Nov 03 01:26:13.725230 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:13.725128    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-885tl"
Nov 03 01:26:13.725230 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:13.725168    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-885tl" event={"ID":"df410080-3e08-4f8d-b1d3-84db851d8782","Type":"ContainerDied","Data":"51ce4bc91216d66f795a38a60af249fef168c1bffc3b3b670aad38674e602941"}
Nov 03 01:26:13.725230 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:13.725227    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-885tl" event={"ID":"df410080-3e08-4f8d-b1d3-84db851d8782","Type":"ContainerDied","Data":"067955b1bf2fea799b4231531f21e8b6c966590e44e3a1e25a3b644dbe8edb31"}
Nov 03 01:26:13.725337 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:13.725250    5012 scope.go:122] "RemoveContainer" containerID="51ce4bc91216d66f795a38a60af249fef168c1bffc3b3b670aad38674e602941"
Nov 03 01:26:13.735566 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:13.735550    5012 scope.go:122] "RemoveContainer" containerID="51ce4bc91216d66f795a38a60af249fef168c1bffc3b3b670aad38674e602941"
Nov 03 01:26:13.735807 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:26:13.735791    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51ce4bc91216d66f795a38a60af249fef168c1bffc3b3b670aad38674e602941\": container with ID starting with 51ce4bc91216d66f795a38a60af249fef168c1bffc3b3b670aad38674e602941 not found: ID does not exist" containerID="51ce4bc91216d66f795a38a60af249fef168c1bffc3b3b670aad38674e602941"
Nov 03 01:26:13.735861 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:13.735815    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ce4bc91216d66f795a38a60af249fef168c1bffc3b3b670aad38674e602941"} err="failed to get container status \"51ce4bc91216d66f795a38a60af249fef168c1bffc3b3b670aad38674e602941\": rpc error: code = NotFound desc = could not find container \"51ce4bc91216d66f795a38a60af249fef168c1bffc3b3b670aad38674e602941\": container with ID starting with 51ce4bc91216d66f795a38a60af249fef168c1bffc3b3b670aad38674e602941 not found: ID does not exist"
Nov 03 01:26:13.770596 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:13.770569    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-885tl"]
Nov 03 01:26:13.773248 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:13.773227    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-885tl"]
Nov 03 01:26:15.242997 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:15.242965    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df410080-3e08-4f8d-b1d3-84db851d8782" path="/var/lib/kubelet/pods/df410080-3e08-4f8d-b1d3-84db851d8782/volumes"
Nov 03 01:26:22.029467 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:22.029429    5012 scope.go:122] "RemoveContainer" containerID="42a6fae3fa6a8f50a3cdcce879d1345dc026e3d5126fcddc081cf55cf2613b9d"
Nov 03 01:26:22.039799 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:22.039774    5012 scope.go:122] "RemoveContainer" containerID="e6032ff8afbd651dc2e0eb6b5b874c052bd74bb6ff148fdf37d5c8cf07c1729c"
Nov 03 01:26:37.361964 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:26:37.361924    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:26:41.038293 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:41.038254    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1479c112-9g6p5"]
Nov 03 01:26:41.048494 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:41.048466    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1479c112-9g6p5"]
Nov 03 01:26:41.242905 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:41.242869    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca" path="/var/lib/kubelet/pods/7ea06228-f4d0-4a50-ab6d-74a5f8dd0cca/volumes"
Nov 03 01:26:57.025925 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:57.025875    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-243c406d-m6ls4"]
Nov 03 01:26:57.035507 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:57.035475    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-243c406d-m6ls4"]
Nov 03 01:26:57.242172 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:26:57.242139    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f359a86-ebbb-4474-bb81-eea347c7ecb5" path="/var/lib/kubelet/pods/5f359a86-ebbb-4474-bb81-eea347c7ecb5/volumes"
Nov 03 01:27:07.362243 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:27:07.362160    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:27:22.075315 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:22.075283    5012 scope.go:122] "RemoveContainer" containerID="d36d2177bc42e4fb6f2479e1f9293af782eeddec756522dbb648e5d1ae7db8c2"
Nov 03 01:27:22.085020 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:22.084999    5012 scope.go:122] "RemoveContainer" containerID="e04713d3495423d2e4a61196dbfbb872bc9704c1db60b2b99f40f61124bcc18b"
Nov 03 01:27:37.362207 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:27:37.362171    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:27:51.650830 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:51.650789    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-cqlms"]
Nov 03 01:27:51.651464 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:51.651441    5012 cpu_manager.go:414] "containerMap: removing container" podUID="0bff2a9d-336f-43b2-ba62-14d7bca26a9a" containerName="revoke-keys"
Nov 03 01:27:51.651563 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:51.651466    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bff2a9d-336f-43b2-ba62-14d7bca26a9a" containerName="revoke-keys"
Nov 03 01:27:51.651563 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:51.651484    5012 cpu_manager.go:414] "containerMap: removing container" podUID="df410080-3e08-4f8d-b1d3-84db851d8782" containerName="payload-processing"
Nov 03 01:27:51.651563 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:51.651493    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df410080-3e08-4f8d-b1d3-84db851d8782" containerName="payload-processing"
Nov 03 01:27:51.651742 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:51.651597    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bff2a9d-336f-43b2-ba62-14d7bca26a9a" containerName="revoke-keys"
Nov 03 01:27:51.651742 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:51.651613    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="df410080-3e08-4f8d-b1d3-84db851d8782" containerName="payload-processing"
Nov 03 01:27:51.655003 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:51.654981    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-cqlms"
Nov 03 01:27:51.673188 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:51.673162    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-cqlms"]
Nov 03 01:27:51.680427 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:51.680406    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:27:51.680544 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:51.680457    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:27:51.680657 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:51.680635    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:27:51.680749 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:51.680714    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:27:51.744407 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:51.744379    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2194817d-464f-47aa-aaad-2b751cd0d2d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-cqlms\" (UID: \"2194817d-464f-47aa-aaad-2b751cd0d2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-cqlms"
Nov 03 01:27:51.744543 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:51.744443    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkhwt\" (UniqueName: \"kubernetes.io/projected/2194817d-464f-47aa-aaad-2b751cd0d2d1-kube-api-access-zkhwt\") pod \"maas-api-revoke-sub-e2547622-cqlms\" (UID: \"2194817d-464f-47aa-aaad-2b751cd0d2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-cqlms"
Nov 03 01:27:51.744543 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:51.744528    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2194817d-464f-47aa-aaad-2b751cd0d2d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-cqlms\" (UID: \"2194817d-464f-47aa-aaad-2b751cd0d2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-cqlms"
Nov 03 01:27:51.845073 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:51.845047    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2194817d-464f-47aa-aaad-2b751cd0d2d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-cqlms\" (UID: \"2194817d-464f-47aa-aaad-2b751cd0d2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-cqlms"
Nov 03 01:27:51.845196 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:51.845092    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkhwt\" (UniqueName: \"kubernetes.io/projected/2194817d-464f-47aa-aaad-2b751cd0d2d1-kube-api-access-zkhwt\") pod \"maas-api-revoke-sub-e2547622-cqlms\" (UID: \"2194817d-464f-47aa-aaad-2b751cd0d2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-cqlms"
Nov 03 01:27:51.845196 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:51.845121    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2194817d-464f-47aa-aaad-2b751cd0d2d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-cqlms\" (UID: \"2194817d-464f-47aa-aaad-2b751cd0d2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-cqlms"
Nov 03 01:27:51.845750 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:51.845728    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2194817d-464f-47aa-aaad-2b751cd0d2d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-cqlms\" (UID: \"2194817d-464f-47aa-aaad-2b751cd0d2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-cqlms"
Nov 03 01:27:51.847442 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:51.847421    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2194817d-464f-47aa-aaad-2b751cd0d2d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-cqlms\" (UID: \"2194817d-464f-47aa-aaad-2b751cd0d2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-cqlms"
Nov 03 01:27:51.862248 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:51.862229    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkhwt\" (UniqueName: \"kubernetes.io/projected/2194817d-464f-47aa-aaad-2b751cd0d2d1-kube-api-access-zkhwt\") pod \"maas-api-revoke-sub-e2547622-cqlms\" (UID: \"2194817d-464f-47aa-aaad-2b751cd0d2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-cqlms"
Nov 03 01:27:51.968444 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:51.968391    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-cqlms"
Nov 03 01:27:52.260451 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:52.260424    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-cqlms"]
Nov 03 01:27:52.261371 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:27:52.261338    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2194817d_464f_47aa_aaad_2b751cd0d2d1.slice/crio-f0496bf97372cfe1957fde5ff655ea2010fe10bca1241ff1e9b52302d1134e01 WatchSource:0}: Error finding container f0496bf97372cfe1957fde5ff655ea2010fe10bca1241ff1e9b52302d1134e01: Status 404 returned error can't find the container with id f0496bf97372cfe1957fde5ff655ea2010fe10bca1241ff1e9b52302d1134e01
Nov 03 01:27:52.355014 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:52.354980    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-cqlms" event={"ID":"2194817d-464f-47aa-aaad-2b751cd0d2d1","Type":"ContainerStarted","Data":"f0496bf97372cfe1957fde5ff655ea2010fe10bca1241ff1e9b52302d1134e01"}
Nov 03 01:27:53.362354 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:53.362322    5012 generic.go:356] "Generic (PLEG): container finished" podID="2194817d-464f-47aa-aaad-2b751cd0d2d1" containerID="31183769b2ecbfcb9429356528f18358bc4c5817259ca6c32d31a987da006466" exitCode=0
Nov 03 01:27:53.362802 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:53.362359    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-cqlms" event={"ID":"2194817d-464f-47aa-aaad-2b751cd0d2d1","Type":"ContainerDied","Data":"31183769b2ecbfcb9429356528f18358bc4c5817259ca6c32d31a987da006466"}
Nov 03 01:27:54.509709 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:54.509685    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-cqlms"
Nov 03 01:27:54.670762 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:54.670684    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2194817d-464f-47aa-aaad-2b751cd0d2d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2194817d-464f-47aa-aaad-2b751cd0d2d1-maas-api-openshift-ca\") pod \"2194817d-464f-47aa-aaad-2b751cd0d2d1\" (UID: \"2194817d-464f-47aa-aaad-2b751cd0d2d1\") "
Nov 03 01:27:54.670902 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:54.670780    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2194817d-464f-47aa-aaad-2b751cd0d2d1-kube-api-access-zkhwt\" (UniqueName: \"kubernetes.io/projected/2194817d-464f-47aa-aaad-2b751cd0d2d1-kube-api-access-zkhwt\") pod \"2194817d-464f-47aa-aaad-2b751cd0d2d1\" (UID: \"2194817d-464f-47aa-aaad-2b751cd0d2d1\") "
Nov 03 01:27:54.670902 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:54.670814    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2194817d-464f-47aa-aaad-2b751cd0d2d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2194817d-464f-47aa-aaad-2b751cd0d2d1-maas-api-xks-ca\") pod \"2194817d-464f-47aa-aaad-2b751cd0d2d1\" (UID: \"2194817d-464f-47aa-aaad-2b751cd0d2d1\") "
Nov 03 01:27:54.671097 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:54.671072    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2194817d-464f-47aa-aaad-2b751cd0d2d1-maas-api-openshift-ca" pod "2194817d-464f-47aa-aaad-2b751cd0d2d1" (UID: "2194817d-464f-47aa-aaad-2b751cd0d2d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:27:54.672779 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:54.672758    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2194817d-464f-47aa-aaad-2b751cd0d2d1-maas-api-xks-ca" pod "2194817d-464f-47aa-aaad-2b751cd0d2d1" (UID: "2194817d-464f-47aa-aaad-2b751cd0d2d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:27:54.672841 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:54.672772    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2194817d-464f-47aa-aaad-2b751cd0d2d1-kube-api-access-zkhwt" pod "2194817d-464f-47aa-aaad-2b751cd0d2d1" (UID: "2194817d-464f-47aa-aaad-2b751cd0d2d1"). InnerVolumeSpecName "kube-api-access-zkhwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:27:54.771746 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:54.771711    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2194817d-464f-47aa-aaad-2b751cd0d2d1-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:27:54.771746 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:54.771746    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkhwt\" (UniqueName: \"kubernetes.io/projected/2194817d-464f-47aa-aaad-2b751cd0d2d1-kube-api-access-zkhwt\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:27:54.771746 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:54.771756    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2194817d-464f-47aa-aaad-2b751cd0d2d1-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:27:55.375447 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:55.375409    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-cqlms" event={"ID":"2194817d-464f-47aa-aaad-2b751cd0d2d1","Type":"ContainerDied","Data":"f0496bf97372cfe1957fde5ff655ea2010fe10bca1241ff1e9b52302d1134e01"}
Nov 03 01:27:55.375621 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:55.375477    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0496bf97372cfe1957fde5ff655ea2010fe10bca1241ff1e9b52302d1134e01"
Nov 03 01:27:55.375621 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:27:55.375421    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-cqlms"
Nov 03 01:28:07.362655 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:28:07.362616    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:28:15.009102 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:15.009065    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlvjt"]
Nov 03 01:28:15.009497 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:15.009473    5012 cpu_manager.go:414] "containerMap: removing container" podUID="2194817d-464f-47aa-aaad-2b751cd0d2d1" containerName="revoke-keys"
Nov 03 01:28:15.009497 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:15.009484    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2194817d-464f-47aa-aaad-2b751cd0d2d1" containerName="revoke-keys"
Nov 03 01:28:15.009583 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:15.009570    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="2194817d-464f-47aa-aaad-2b751cd0d2d1" containerName="revoke-keys"
Nov 03 01:28:15.013082 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:15.013065    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlvjt"
Nov 03 01:28:15.030388 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:15.030356    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlvjt"]
Nov 03 01:28:15.037433 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:15.037403    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:28:15.037558 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:15.037415    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:28:15.037558 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:15.037434    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:28:15.037558 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:15.037468    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:28:15.081799 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:15.081775    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/869b15c7-0285-4c63-b2d3-3cd538662b56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-nlvjt\" (UID: \"869b15c7-0285-4c63-b2d3-3cd538662b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlvjt"
Nov 03 01:28:15.081934 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:15.081812    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpw8l\" (UniqueName: \"kubernetes.io/projected/869b15c7-0285-4c63-b2d3-3cd538662b56-kube-api-access-kpw8l\") pod \"maas-api-revoke-sub-c1e271a9-nlvjt\" (UID: \"869b15c7-0285-4c63-b2d3-3cd538662b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlvjt"
Nov 03 01:28:15.081988 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:15.081962    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/869b15c7-0285-4c63-b2d3-3cd538662b56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-nlvjt\" (UID: \"869b15c7-0285-4c63-b2d3-3cd538662b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlvjt"
Nov 03 01:28:15.183206 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:15.183185    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/869b15c7-0285-4c63-b2d3-3cd538662b56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-nlvjt\" (UID: \"869b15c7-0285-4c63-b2d3-3cd538662b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlvjt"
Nov 03 01:28:15.183317 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:15.183235    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/869b15c7-0285-4c63-b2d3-3cd538662b56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-nlvjt\" (UID: \"869b15c7-0285-4c63-b2d3-3cd538662b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlvjt"
Nov 03 01:28:15.183317 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:15.183262    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpw8l\" (UniqueName: \"kubernetes.io/projected/869b15c7-0285-4c63-b2d3-3cd538662b56-kube-api-access-kpw8l\") pod \"maas-api-revoke-sub-c1e271a9-nlvjt\" (UID: \"869b15c7-0285-4c63-b2d3-3cd538662b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlvjt"
Nov 03 01:28:15.183957 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:15.183902    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/869b15c7-0285-4c63-b2d3-3cd538662b56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-nlvjt\" (UID: \"869b15c7-0285-4c63-b2d3-3cd538662b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlvjt"
Nov 03 01:28:15.185536 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:15.185515    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/869b15c7-0285-4c63-b2d3-3cd538662b56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-nlvjt\" (UID: \"869b15c7-0285-4c63-b2d3-3cd538662b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlvjt"
Nov 03 01:28:15.201129 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:15.201102    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpw8l\" (UniqueName: \"kubernetes.io/projected/869b15c7-0285-4c63-b2d3-3cd538662b56-kube-api-access-kpw8l\") pod \"maas-api-revoke-sub-c1e271a9-nlvjt\" (UID: \"869b15c7-0285-4c63-b2d3-3cd538662b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlvjt"
Nov 03 01:28:15.326385 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:15.326329    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlvjt"
Nov 03 01:28:15.619231 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:15.619201    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlvjt"]
Nov 03 01:28:15.620530 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:28:15.620504    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod869b15c7_0285_4c63_b2d3_3cd538662b56.slice/crio-964a2d8187c659c3dfb3c8ddf49cd6dd179d12ea50e89939f1057790559a367e WatchSource:0}: Error finding container 964a2d8187c659c3dfb3c8ddf49cd6dd179d12ea50e89939f1057790559a367e: Status 404 returned error can't find the container with id 964a2d8187c659c3dfb3c8ddf49cd6dd179d12ea50e89939f1057790559a367e
Nov 03 01:28:16.545543 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:16.545512    5012 generic.go:356] "Generic (PLEG): container finished" podID="869b15c7-0285-4c63-b2d3-3cd538662b56" containerID="2b9b1b2a70b544674166e7b3aa15fff64429c2af7ecfaa8f352362353ae773bd" exitCode=0
Nov 03 01:28:16.545864 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:16.545592    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlvjt" event={"ID":"869b15c7-0285-4c63-b2d3-3cd538662b56","Type":"ContainerDied","Data":"2b9b1b2a70b544674166e7b3aa15fff64429c2af7ecfaa8f352362353ae773bd"}
Nov 03 01:28:16.545864 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:16.545653    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlvjt" event={"ID":"869b15c7-0285-4c63-b2d3-3cd538662b56","Type":"ContainerStarted","Data":"964a2d8187c659c3dfb3c8ddf49cd6dd179d12ea50e89939f1057790559a367e"}
Nov 03 01:28:17.699270 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:17.699246    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlvjt"
Nov 03 01:28:17.699770 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:17.699752    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/869b15c7-0285-4c63-b2d3-3cd538662b56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/869b15c7-0285-4c63-b2d3-3cd538662b56-maas-api-openshift-ca\") pod \"869b15c7-0285-4c63-b2d3-3cd538662b56\" (UID: \"869b15c7-0285-4c63-b2d3-3cd538662b56\") "
Nov 03 01:28:17.699835 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:17.699821    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/869b15c7-0285-4c63-b2d3-3cd538662b56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/869b15c7-0285-4c63-b2d3-3cd538662b56-maas-api-xks-ca\") pod \"869b15c7-0285-4c63-b2d3-3cd538662b56\" (UID: \"869b15c7-0285-4c63-b2d3-3cd538662b56\") "
Nov 03 01:28:17.699892 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:17.699875    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/869b15c7-0285-4c63-b2d3-3cd538662b56-kube-api-access-kpw8l\" (UniqueName: \"kubernetes.io/projected/869b15c7-0285-4c63-b2d3-3cd538662b56-kube-api-access-kpw8l\") pod \"869b15c7-0285-4c63-b2d3-3cd538662b56\" (UID: \"869b15c7-0285-4c63-b2d3-3cd538662b56\") "
Nov 03 01:28:17.700093 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:17.700070    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/869b15c7-0285-4c63-b2d3-3cd538662b56-maas-api-openshift-ca" pod "869b15c7-0285-4c63-b2d3-3cd538662b56" (UID: "869b15c7-0285-4c63-b2d3-3cd538662b56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:28:17.700333 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:17.700304    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/869b15c7-0285-4c63-b2d3-3cd538662b56-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:28:17.701786 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:17.701759    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/869b15c7-0285-4c63-b2d3-3cd538662b56-maas-api-xks-ca" pod "869b15c7-0285-4c63-b2d3-3cd538662b56" (UID: "869b15c7-0285-4c63-b2d3-3cd538662b56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:28:17.702006 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:17.701984    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/869b15c7-0285-4c63-b2d3-3cd538662b56-kube-api-access-kpw8l" pod "869b15c7-0285-4c63-b2d3-3cd538662b56" (UID: "869b15c7-0285-4c63-b2d3-3cd538662b56"). InnerVolumeSpecName "kube-api-access-kpw8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:28:17.800856 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:17.800826    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/869b15c7-0285-4c63-b2d3-3cd538662b56-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:28:17.800856 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:17.800852    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpw8l\" (UniqueName: \"kubernetes.io/projected/869b15c7-0285-4c63-b2d3-3cd538662b56-kube-api-access-kpw8l\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:28:18.559501 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:18.559459    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlvjt" event={"ID":"869b15c7-0285-4c63-b2d3-3cd538662b56","Type":"ContainerDied","Data":"964a2d8187c659c3dfb3c8ddf49cd6dd179d12ea50e89939f1057790559a367e"}
Nov 03 01:28:18.559714 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:18.559526    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="964a2d8187c659c3dfb3c8ddf49cd6dd179d12ea50e89939f1057790559a367e"
Nov 03 01:28:18.559714 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:18.559472    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlvjt"
Nov 03 01:28:22.111301 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:22.111270    5012 scope.go:122] "RemoveContainer" containerID="7fd2af65d7bae9fa7aea3e8014dde362b0874e3449cf4fe999f33605a3837541"
Nov 03 01:28:22.120427 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:22.120406    5012 scope.go:122] "RemoveContainer" containerID="7acafd912b59d41dbba3350b1dc1b859f88ce9850f26a91f4e9675fe5c1b68fa"
Nov 03 01:28:34.491973 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:34.491863    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fkgtf"]
Nov 03 01:28:34.492470 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:34.492320    5012 cpu_manager.go:414] "containerMap: removing container" podUID="869b15c7-0285-4c63-b2d3-3cd538662b56" containerName="revoke-keys"
Nov 03 01:28:34.492470 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:34.492333    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="869b15c7-0285-4c63-b2d3-3cd538662b56" containerName="revoke-keys"
Nov 03 01:28:34.492470 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:34.492406    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="869b15c7-0285-4c63-b2d3-3cd538662b56" containerName="revoke-keys"
Nov 03 01:28:34.495569 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:34.495552    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fkgtf"
Nov 03 01:28:34.512713 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:34.512688    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fkgtf"]
Nov 03 01:28:34.520942 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:34.519378    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:28:34.520942 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:34.519496    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:28:34.520942 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:34.519529    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:28:34.520942 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:34.519557    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:28:34.577561 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:34.577531    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdnqn\" (UniqueName: \"kubernetes.io/projected/6f193d66-6bbe-4c27-b9f0-0df1f0605af1-kube-api-access-hdnqn\") pod \"maas-api-revoke-sub-2e1f5b5f-fkgtf\" (UID: \"6f193d66-6bbe-4c27-b9f0-0df1f0605af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fkgtf"
Nov 03 01:28:34.577686 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:34.577593    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f193d66-6bbe-4c27-b9f0-0df1f0605af1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-fkgtf\" (UID: \"6f193d66-6bbe-4c27-b9f0-0df1f0605af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fkgtf"
Nov 03 01:28:34.577737 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:34.577695    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f193d66-6bbe-4c27-b9f0-0df1f0605af1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-fkgtf\" (UID: \"6f193d66-6bbe-4c27-b9f0-0df1f0605af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fkgtf"
Nov 03 01:28:34.678863 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:34.678832    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f193d66-6bbe-4c27-b9f0-0df1f0605af1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-fkgtf\" (UID: \"6f193d66-6bbe-4c27-b9f0-0df1f0605af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fkgtf"
Nov 03 01:28:34.679066 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:34.678893    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f193d66-6bbe-4c27-b9f0-0df1f0605af1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-fkgtf\" (UID: \"6f193d66-6bbe-4c27-b9f0-0df1f0605af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fkgtf"
Nov 03 01:28:34.679066 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:34.678984    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdnqn\" (UniqueName: \"kubernetes.io/projected/6f193d66-6bbe-4c27-b9f0-0df1f0605af1-kube-api-access-hdnqn\") pod \"maas-api-revoke-sub-2e1f5b5f-fkgtf\" (UID: \"6f193d66-6bbe-4c27-b9f0-0df1f0605af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fkgtf"
Nov 03 01:28:34.679625 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:34.679600    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f193d66-6bbe-4c27-b9f0-0df1f0605af1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-fkgtf\" (UID: \"6f193d66-6bbe-4c27-b9f0-0df1f0605af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fkgtf"
Nov 03 01:28:34.681153 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:34.681133    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f193d66-6bbe-4c27-b9f0-0df1f0605af1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-fkgtf\" (UID: \"6f193d66-6bbe-4c27-b9f0-0df1f0605af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fkgtf"
Nov 03 01:28:34.695831 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:34.695806    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdnqn\" (UniqueName: \"kubernetes.io/projected/6f193d66-6bbe-4c27-b9f0-0df1f0605af1-kube-api-access-hdnqn\") pod \"maas-api-revoke-sub-2e1f5b5f-fkgtf\" (UID: \"6f193d66-6bbe-4c27-b9f0-0df1f0605af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fkgtf"
Nov 03 01:28:34.809188 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:34.809135    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fkgtf"
Nov 03 01:28:35.102855 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:35.102824    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fkgtf"]
Nov 03 01:28:35.104333 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:28:35.104302    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f193d66_6bbe_4c27_b9f0_0df1f0605af1.slice/crio-7ceac37e44706c9560994827bd8cf7a018c7ad4213aaad164a4fb81fa80f59d3 WatchSource:0}: Error finding container 7ceac37e44706c9560994827bd8cf7a018c7ad4213aaad164a4fb81fa80f59d3: Status 404 returned error can't find the container with id 7ceac37e44706c9560994827bd8cf7a018c7ad4213aaad164a4fb81fa80f59d3
Nov 03 01:28:35.683221 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:35.683143    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fkgtf" event={"ID":"6f193d66-6bbe-4c27-b9f0-0df1f0605af1","Type":"ContainerStarted","Data":"9c36e28cdc158e1dcea20e7170eee05d6701355c9e10ce83fab8530730784d2b"}
Nov 03 01:28:35.683221 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:35.683197    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fkgtf" event={"ID":"6f193d66-6bbe-4c27-b9f0-0df1f0605af1","Type":"ContainerStarted","Data":"7ceac37e44706c9560994827bd8cf7a018c7ad4213aaad164a4fb81fa80f59d3"}
Nov 03 01:28:35.718247 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:35.718195    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fkgtf" podStartSLOduration=1.7181815120000001 podStartE2EDuration="1.718181512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:35.716938083 +0000 UTC m=+4998.596522118" watchObservedRunningTime="2026-11-03 01:28:35.718181512 +0000 UTC m=+4998.597765555"
Nov 03 01:28:36.690497 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:36.690462    5012 generic.go:356] "Generic (PLEG): container finished" podID="6f193d66-6bbe-4c27-b9f0-0df1f0605af1" containerID="9c36e28cdc158e1dcea20e7170eee05d6701355c9e10ce83fab8530730784d2b" exitCode=0
Nov 03 01:28:36.690877 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:36.690551    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fkgtf" event={"ID":"6f193d66-6bbe-4c27-b9f0-0df1f0605af1","Type":"ContainerDied","Data":"9c36e28cdc158e1dcea20e7170eee05d6701355c9e10ce83fab8530730784d2b"}
Nov 03 01:28:37.362305 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:28:37.362273    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:28:37.830633 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:37.830609    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fkgtf"
Nov 03 01:28:37.910050 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:37.910023    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f193d66-6bbe-4c27-b9f0-0df1f0605af1-kube-api-access-hdnqn\" (UniqueName: \"kubernetes.io/projected/6f193d66-6bbe-4c27-b9f0-0df1f0605af1-kube-api-access-hdnqn\") pod \"6f193d66-6bbe-4c27-b9f0-0df1f0605af1\" (UID: \"6f193d66-6bbe-4c27-b9f0-0df1f0605af1\") "
Nov 03 01:28:37.910203 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:37.910136    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f193d66-6bbe-4c27-b9f0-0df1f0605af1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f193d66-6bbe-4c27-b9f0-0df1f0605af1-maas-api-openshift-ca\") pod \"6f193d66-6bbe-4c27-b9f0-0df1f0605af1\" (UID: \"6f193d66-6bbe-4c27-b9f0-0df1f0605af1\") "
Nov 03 01:28:37.910203 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:37.910170    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f193d66-6bbe-4c27-b9f0-0df1f0605af1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f193d66-6bbe-4c27-b9f0-0df1f0605af1-maas-api-xks-ca\") pod \"6f193d66-6bbe-4c27-b9f0-0df1f0605af1\" (UID: \"6f193d66-6bbe-4c27-b9f0-0df1f0605af1\") "
Nov 03 01:28:37.910538 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:37.910512    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f193d66-6bbe-4c27-b9f0-0df1f0605af1-maas-api-openshift-ca" pod "6f193d66-6bbe-4c27-b9f0-0df1f0605af1" (UID: "6f193d66-6bbe-4c27-b9f0-0df1f0605af1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:28:37.912077 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:37.912054    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f193d66-6bbe-4c27-b9f0-0df1f0605af1-kube-api-access-hdnqn" pod "6f193d66-6bbe-4c27-b9f0-0df1f0605af1" (UID: "6f193d66-6bbe-4c27-b9f0-0df1f0605af1"). InnerVolumeSpecName "kube-api-access-hdnqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:28:37.912180 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:37.912140    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f193d66-6bbe-4c27-b9f0-0df1f0605af1-maas-api-xks-ca" pod "6f193d66-6bbe-4c27-b9f0-0df1f0605af1" (UID: "6f193d66-6bbe-4c27-b9f0-0df1f0605af1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:28:38.011421 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:38.011369    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f193d66-6bbe-4c27-b9f0-0df1f0605af1-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:28:38.011421 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:38.011391    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f193d66-6bbe-4c27-b9f0-0df1f0605af1-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:28:38.011421 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:38.011400    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdnqn\" (UniqueName: \"kubernetes.io/projected/6f193d66-6bbe-4c27-b9f0-0df1f0605af1-kube-api-access-hdnqn\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:28:38.703113 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:38.703074    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fkgtf" event={"ID":"6f193d66-6bbe-4c27-b9f0-0df1f0605af1","Type":"ContainerDied","Data":"7ceac37e44706c9560994827bd8cf7a018c7ad4213aaad164a4fb81fa80f59d3"}
Nov 03 01:28:38.703113 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:38.703111    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fkgtf"
Nov 03 01:28:38.703320 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:38.703136    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ceac37e44706c9560994827bd8cf7a018c7ad4213aaad164a4fb81fa80f59d3"
Nov 03 01:28:45.838821 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:45.838785    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x98rl"]
Nov 03 01:28:45.839439 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:45.839219    5012 cpu_manager.go:414] "containerMap: removing container" podUID="6f193d66-6bbe-4c27-b9f0-0df1f0605af1" containerName="revoke-keys"
Nov 03 01:28:45.839439 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:45.839231    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f193d66-6bbe-4c27-b9f0-0df1f0605af1" containerName="revoke-keys"
Nov 03 01:28:45.839439 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:45.839311    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f193d66-6bbe-4c27-b9f0-0df1f0605af1" containerName="revoke-keys"
Nov 03 01:28:45.842778 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:45.842755    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x98rl"
Nov 03 01:28:45.856877 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:45.856854    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:28:45.857022 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:45.856939    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:28:45.857022 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:45.856951    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:28:45.857022 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:45.856967    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:28:45.872238 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:45.872207    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x98rl"]
Nov 03 01:28:45.880386 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:45.880356    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82206537-1719-4b8c-896e-6d84a680f7af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-x98rl\" (UID: \"82206537-1719-4b8c-896e-6d84a680f7af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x98rl"
Nov 03 01:28:45.880556 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:45.880461    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82206537-1719-4b8c-896e-6d84a680f7af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-x98rl\" (UID: \"82206537-1719-4b8c-896e-6d84a680f7af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x98rl"
Nov 03 01:28:45.880556 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:45.880490    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm9dc\" (UniqueName: \"kubernetes.io/projected/82206537-1719-4b8c-896e-6d84a680f7af-kube-api-access-rm9dc\") pod \"maas-api-revoke-sub-ec843018-x98rl\" (UID: \"82206537-1719-4b8c-896e-6d84a680f7af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x98rl"
Nov 03 01:28:45.981902 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:45.981865    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82206537-1719-4b8c-896e-6d84a680f7af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-x98rl\" (UID: \"82206537-1719-4b8c-896e-6d84a680f7af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x98rl"
Nov 03 01:28:45.982091 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:45.981993    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82206537-1719-4b8c-896e-6d84a680f7af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-x98rl\" (UID: \"82206537-1719-4b8c-896e-6d84a680f7af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x98rl"
Nov 03 01:28:45.982091 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:45.982023    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm9dc\" (UniqueName: \"kubernetes.io/projected/82206537-1719-4b8c-896e-6d84a680f7af-kube-api-access-rm9dc\") pod \"maas-api-revoke-sub-ec843018-x98rl\" (UID: \"82206537-1719-4b8c-896e-6d84a680f7af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x98rl"
Nov 03 01:28:45.982670 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:45.982647    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82206537-1719-4b8c-896e-6d84a680f7af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-x98rl\" (UID: \"82206537-1719-4b8c-896e-6d84a680f7af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x98rl"
Nov 03 01:28:45.984187 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:45.984164    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82206537-1719-4b8c-896e-6d84a680f7af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-x98rl\" (UID: \"82206537-1719-4b8c-896e-6d84a680f7af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x98rl"
Nov 03 01:28:45.998834 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:45.998809    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm9dc\" (UniqueName: \"kubernetes.io/projected/82206537-1719-4b8c-896e-6d84a680f7af-kube-api-access-rm9dc\") pod \"maas-api-revoke-sub-ec843018-x98rl\" (UID: \"82206537-1719-4b8c-896e-6d84a680f7af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x98rl"
Nov 03 01:28:46.169702 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:46.169659    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x98rl"
Nov 03 01:28:46.499172 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:46.499142    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x98rl"]
Nov 03 01:28:46.501077 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:28:46.501050    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82206537_1719_4b8c_896e_6d84a680f7af.slice/crio-69b953324bc8de92b6a9d3b1cc7d9613bebaf42fdcd289f5ee7c4972be6b9a8b WatchSource:0}: Error finding container 69b953324bc8de92b6a9d3b1cc7d9613bebaf42fdcd289f5ee7c4972be6b9a8b: Status 404 returned error can't find the container with id 69b953324bc8de92b6a9d3b1cc7d9613bebaf42fdcd289f5ee7c4972be6b9a8b
Nov 03 01:28:46.751575 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:46.751514    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x98rl" event={"ID":"82206537-1719-4b8c-896e-6d84a680f7af","Type":"ContainerStarted","Data":"69b953324bc8de92b6a9d3b1cc7d9613bebaf42fdcd289f5ee7c4972be6b9a8b"}
Nov 03 01:28:47.759437 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:47.759408    5012 generic.go:356] "Generic (PLEG): container finished" podID="82206537-1719-4b8c-896e-6d84a680f7af" containerID="10348978cf354c0f8003c0d8e9c9acd5314ee93769c3370c317916ff17cb0d91" exitCode=0
Nov 03 01:28:47.759817 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:47.759487    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x98rl" event={"ID":"82206537-1719-4b8c-896e-6d84a680f7af","Type":"ContainerDied","Data":"10348978cf354c0f8003c0d8e9c9acd5314ee93769c3370c317916ff17cb0d91"}
Nov 03 01:28:48.910979 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:48.910946    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x98rl"
Nov 03 01:28:49.008979 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:49.008947    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82206537-1719-4b8c-896e-6d84a680f7af-kube-api-access-rm9dc\" (UniqueName: \"kubernetes.io/projected/82206537-1719-4b8c-896e-6d84a680f7af-kube-api-access-rm9dc\") pod \"82206537-1719-4b8c-896e-6d84a680f7af\" (UID: \"82206537-1719-4b8c-896e-6d84a680f7af\") "
Nov 03 01:28:49.009141 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:49.009000    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82206537-1719-4b8c-896e-6d84a680f7af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82206537-1719-4b8c-896e-6d84a680f7af-maas-api-openshift-ca\") pod \"82206537-1719-4b8c-896e-6d84a680f7af\" (UID: \"82206537-1719-4b8c-896e-6d84a680f7af\") "
Nov 03 01:28:49.009141 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:49.009127    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82206537-1719-4b8c-896e-6d84a680f7af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82206537-1719-4b8c-896e-6d84a680f7af-maas-api-xks-ca\") pod \"82206537-1719-4b8c-896e-6d84a680f7af\" (UID: \"82206537-1719-4b8c-896e-6d84a680f7af\") "
Nov 03 01:28:49.009346 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:49.009323    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82206537-1719-4b8c-896e-6d84a680f7af-maas-api-openshift-ca" pod "82206537-1719-4b8c-896e-6d84a680f7af" (UID: "82206537-1719-4b8c-896e-6d84a680f7af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:28:49.010991 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:49.010963    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82206537-1719-4b8c-896e-6d84a680f7af-maas-api-xks-ca" pod "82206537-1719-4b8c-896e-6d84a680f7af" (UID: "82206537-1719-4b8c-896e-6d84a680f7af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:28:49.010991 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:49.010963    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82206537-1719-4b8c-896e-6d84a680f7af-kube-api-access-rm9dc" pod "82206537-1719-4b8c-896e-6d84a680f7af" (UID: "82206537-1719-4b8c-896e-6d84a680f7af"). InnerVolumeSpecName "kube-api-access-rm9dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:28:49.110716 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:49.110690    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm9dc\" (UniqueName: \"kubernetes.io/projected/82206537-1719-4b8c-896e-6d84a680f7af-kube-api-access-rm9dc\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:28:49.110716 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:49.110715    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82206537-1719-4b8c-896e-6d84a680f7af-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:28:49.110846 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:49.110725    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82206537-1719-4b8c-896e-6d84a680f7af-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:28:49.773473 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:49.773441    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x98rl"
Nov 03 01:28:49.773690 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:49.773470    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x98rl" event={"ID":"82206537-1719-4b8c-896e-6d84a680f7af","Type":"ContainerDied","Data":"69b953324bc8de92b6a9d3b1cc7d9613bebaf42fdcd289f5ee7c4972be6b9a8b"}
Nov 03 01:28:49.773690 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:28:49.773524    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69b953324bc8de92b6a9d3b1cc7d9613bebaf42fdcd289f5ee7c4972be6b9a8b"
Nov 03 01:29:07.362519 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:29:07.362482    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:29:12.820683 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:12.820648    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8sjj9"]
Nov 03 01:29:12.821132 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:12.821116    5012 cpu_manager.go:414] "containerMap: removing container" podUID="82206537-1719-4b8c-896e-6d84a680f7af" containerName="revoke-keys"
Nov 03 01:29:12.821181 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:12.821132    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82206537-1719-4b8c-896e-6d84a680f7af" containerName="revoke-keys"
Nov 03 01:29:12.821219 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:12.821207    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="82206537-1719-4b8c-896e-6d84a680f7af" containerName="revoke-keys"
Nov 03 01:29:12.824511 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:12.824485    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8sjj9"
Nov 03 01:29:12.832408 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:12.832379    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48fef83-16a3-45bd-ac50-108ba16bb4ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-8sjj9\" (UID: \"d48fef83-16a3-45bd-ac50-108ba16bb4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8sjj9"
Nov 03 01:29:12.832520 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:12.832420    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkvgw\" (UniqueName: \"kubernetes.io/projected/d48fef83-16a3-45bd-ac50-108ba16bb4ec-kube-api-access-bkvgw\") pod \"maas-api-revoke-sub-046e1343-8sjj9\" (UID: \"d48fef83-16a3-45bd-ac50-108ba16bb4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8sjj9"
Nov 03 01:29:12.832595 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:12.832542    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48fef83-16a3-45bd-ac50-108ba16bb4ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-8sjj9\" (UID: \"d48fef83-16a3-45bd-ac50-108ba16bb4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8sjj9"
Nov 03 01:29:12.851148 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:12.851120    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:29:12.851327 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:12.851142    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:29:12.851327 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:12.851191    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:29:12.851327 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:12.851214    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:29:12.855931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:12.855893    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8sjj9"]
Nov 03 01:29:12.933328 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:12.933293    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48fef83-16a3-45bd-ac50-108ba16bb4ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-8sjj9\" (UID: \"d48fef83-16a3-45bd-ac50-108ba16bb4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8sjj9"
Nov 03 01:29:12.933511 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:12.933339    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkvgw\" (UniqueName: \"kubernetes.io/projected/d48fef83-16a3-45bd-ac50-108ba16bb4ec-kube-api-access-bkvgw\") pod \"maas-api-revoke-sub-046e1343-8sjj9\" (UID: \"d48fef83-16a3-45bd-ac50-108ba16bb4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8sjj9"
Nov 03 01:29:12.933511 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:12.933383    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48fef83-16a3-45bd-ac50-108ba16bb4ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-8sjj9\" (UID: \"d48fef83-16a3-45bd-ac50-108ba16bb4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8sjj9"
Nov 03 01:29:12.934056 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:12.934034    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48fef83-16a3-45bd-ac50-108ba16bb4ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-8sjj9\" (UID: \"d48fef83-16a3-45bd-ac50-108ba16bb4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8sjj9"
Nov 03 01:29:12.935672 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:12.935647    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48fef83-16a3-45bd-ac50-108ba16bb4ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-8sjj9\" (UID: \"d48fef83-16a3-45bd-ac50-108ba16bb4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8sjj9"
Nov 03 01:29:12.950518 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:12.950494    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkvgw\" (UniqueName: \"kubernetes.io/projected/d48fef83-16a3-45bd-ac50-108ba16bb4ec-kube-api-access-bkvgw\") pod \"maas-api-revoke-sub-046e1343-8sjj9\" (UID: \"d48fef83-16a3-45bd-ac50-108ba16bb4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8sjj9"
Nov 03 01:29:13.138120 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:13.138093    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8sjj9"
Nov 03 01:29:13.430578 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:13.430551    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8sjj9"]
Nov 03 01:29:13.431796 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:29:13.431770    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48fef83_16a3_45bd_ac50_108ba16bb4ec.slice/crio-ae4ccd299244d636b774ff3aa07a2a308c710176ae5ec67103847360d28a666f WatchSource:0}: Error finding container ae4ccd299244d636b774ff3aa07a2a308c710176ae5ec67103847360d28a666f: Status 404 returned error can't find the container with id ae4ccd299244d636b774ff3aa07a2a308c710176ae5ec67103847360d28a666f
Nov 03 01:29:13.928885 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:13.928843    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8sjj9" event={"ID":"d48fef83-16a3-45bd-ac50-108ba16bb4ec","Type":"ContainerStarted","Data":"3e9cb645c81ac757ccf885f79f44518b66ad1c37ea3ae888f65981024e4ed8fd"}
Nov 03 01:29:13.929345 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:13.928934    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8sjj9" event={"ID":"d48fef83-16a3-45bd-ac50-108ba16bb4ec","Type":"ContainerStarted","Data":"ae4ccd299244d636b774ff3aa07a2a308c710176ae5ec67103847360d28a666f"}
Nov 03 01:29:13.966824 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:13.966738    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8sjj9" podStartSLOduration=1.966724971 podStartE2EDuration="1.966724971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:13.963311423 +0000 UTC m=+5036.842895466" watchObservedRunningTime="2026-11-03 01:29:13.966724971 +0000 UTC m=+5036.846309014"
Nov 03 01:29:14.936545 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:14.936502    5012 generic.go:356] "Generic (PLEG): container finished" podID="d48fef83-16a3-45bd-ac50-108ba16bb4ec" containerID="3e9cb645c81ac757ccf885f79f44518b66ad1c37ea3ae888f65981024e4ed8fd" exitCode=0
Nov 03 01:29:14.937022 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:14.936588    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8sjj9" event={"ID":"d48fef83-16a3-45bd-ac50-108ba16bb4ec","Type":"ContainerDied","Data":"3e9cb645c81ac757ccf885f79f44518b66ad1c37ea3ae888f65981024e4ed8fd"}
Nov 03 01:29:16.081266 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:16.081239    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8sjj9"
Nov 03 01:29:16.156285 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:16.156261    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d48fef83-16a3-45bd-ac50-108ba16bb4ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48fef83-16a3-45bd-ac50-108ba16bb4ec-maas-api-xks-ca\") pod \"d48fef83-16a3-45bd-ac50-108ba16bb4ec\" (UID: \"d48fef83-16a3-45bd-ac50-108ba16bb4ec\") "
Nov 03 01:29:16.156408 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:16.156296    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d48fef83-16a3-45bd-ac50-108ba16bb4ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48fef83-16a3-45bd-ac50-108ba16bb4ec-maas-api-openshift-ca\") pod \"d48fef83-16a3-45bd-ac50-108ba16bb4ec\" (UID: \"d48fef83-16a3-45bd-ac50-108ba16bb4ec\") "
Nov 03 01:29:16.156705 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:16.156676    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d48fef83-16a3-45bd-ac50-108ba16bb4ec-maas-api-openshift-ca" pod "d48fef83-16a3-45bd-ac50-108ba16bb4ec" (UID: "d48fef83-16a3-45bd-ac50-108ba16bb4ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:29:16.158280 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:16.158258    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d48fef83-16a3-45bd-ac50-108ba16bb4ec-maas-api-xks-ca" pod "d48fef83-16a3-45bd-ac50-108ba16bb4ec" (UID: "d48fef83-16a3-45bd-ac50-108ba16bb4ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:29:16.257343 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:16.257274    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d48fef83-16a3-45bd-ac50-108ba16bb4ec-kube-api-access-bkvgw\" (UniqueName: \"kubernetes.io/projected/d48fef83-16a3-45bd-ac50-108ba16bb4ec-kube-api-access-bkvgw\") pod \"d48fef83-16a3-45bd-ac50-108ba16bb4ec\" (UID: \"d48fef83-16a3-45bd-ac50-108ba16bb4ec\") "
Nov 03 01:29:16.257493 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:16.257479    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48fef83-16a3-45bd-ac50-108ba16bb4ec-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:29:16.257493 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:16.257491    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48fef83-16a3-45bd-ac50-108ba16bb4ec-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:29:16.259328 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:16.259298    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48fef83-16a3-45bd-ac50-108ba16bb4ec-kube-api-access-bkvgw" pod "d48fef83-16a3-45bd-ac50-108ba16bb4ec" (UID: "d48fef83-16a3-45bd-ac50-108ba16bb4ec"). InnerVolumeSpecName "kube-api-access-bkvgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:29:16.358073 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:16.358045    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkvgw\" (UniqueName: \"kubernetes.io/projected/d48fef83-16a3-45bd-ac50-108ba16bb4ec-kube-api-access-bkvgw\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:29:16.949978 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:16.949947    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8sjj9"
Nov 03 01:29:16.950245 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:16.949953    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8sjj9" event={"ID":"d48fef83-16a3-45bd-ac50-108ba16bb4ec","Type":"ContainerDied","Data":"ae4ccd299244d636b774ff3aa07a2a308c710176ae5ec67103847360d28a666f"}
Nov 03 01:29:16.950245 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:16.950092    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae4ccd299244d636b774ff3aa07a2a308c710176ae5ec67103847360d28a666f"
Nov 03 01:29:23.171692 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:23.171656    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d68kq"]
Nov 03 01:29:23.172136 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:23.172075    5012 cpu_manager.go:414] "containerMap: removing container" podUID="d48fef83-16a3-45bd-ac50-108ba16bb4ec" containerName="revoke-keys"
Nov 03 01:29:23.172136 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:23.172087    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d48fef83-16a3-45bd-ac50-108ba16bb4ec" containerName="revoke-keys"
Nov 03 01:29:23.172218 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:23.172178    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="d48fef83-16a3-45bd-ac50-108ba16bb4ec" containerName="revoke-keys"
Nov 03 01:29:23.175552 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:23.175534    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d68kq"
Nov 03 01:29:23.189648 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:23.189624    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:29:23.199788 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:23.199752    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:29:23.200096 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:23.200072    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:29:23.200232 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:23.200105    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:29:23.205341 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:23.205315    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d68kq"]
Nov 03 01:29:23.217719 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:23.217696    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqwc5\" (UniqueName: \"kubernetes.io/projected/c8b8a3de-cc6b-489f-8a1d-8be74509e423-kube-api-access-mqwc5\") pod \"maas-api-revoke-sub-d3fba056-d68kq\" (UID: \"c8b8a3de-cc6b-489f-8a1d-8be74509e423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d68kq"
Nov 03 01:29:23.217817 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:23.217732    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b8a3de-cc6b-489f-8a1d-8be74509e423-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-d68kq\" (UID: \"c8b8a3de-cc6b-489f-8a1d-8be74509e423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d68kq"
Nov 03 01:29:23.217868 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:23.217840    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b8a3de-cc6b-489f-8a1d-8be74509e423-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-d68kq\" (UID: \"c8b8a3de-cc6b-489f-8a1d-8be74509e423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d68kq"
Nov 03 01:29:23.319279 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:23.319251    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqwc5\" (UniqueName: \"kubernetes.io/projected/c8b8a3de-cc6b-489f-8a1d-8be74509e423-kube-api-access-mqwc5\") pod \"maas-api-revoke-sub-d3fba056-d68kq\" (UID: \"c8b8a3de-cc6b-489f-8a1d-8be74509e423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d68kq"
Nov 03 01:29:23.319383 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:23.319289    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b8a3de-cc6b-489f-8a1d-8be74509e423-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-d68kq\" (UID: \"c8b8a3de-cc6b-489f-8a1d-8be74509e423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d68kq"
Nov 03 01:29:23.319383 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:23.319343    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b8a3de-cc6b-489f-8a1d-8be74509e423-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-d68kq\" (UID: \"c8b8a3de-cc6b-489f-8a1d-8be74509e423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d68kq"
Nov 03 01:29:23.320106 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:23.320078    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b8a3de-cc6b-489f-8a1d-8be74509e423-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-d68kq\" (UID: \"c8b8a3de-cc6b-489f-8a1d-8be74509e423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d68kq"
Nov 03 01:29:23.321646 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:23.321625    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b8a3de-cc6b-489f-8a1d-8be74509e423-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-d68kq\" (UID: \"c8b8a3de-cc6b-489f-8a1d-8be74509e423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d68kq"
Nov 03 01:29:23.337035 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:23.337008    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqwc5\" (UniqueName: \"kubernetes.io/projected/c8b8a3de-cc6b-489f-8a1d-8be74509e423-kube-api-access-mqwc5\") pod \"maas-api-revoke-sub-d3fba056-d68kq\" (UID: \"c8b8a3de-cc6b-489f-8a1d-8be74509e423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d68kq"
Nov 03 01:29:23.488382 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:23.488322    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d68kq"
Nov 03 01:29:23.781697 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:23.781666    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d68kq"]
Nov 03 01:29:23.782958 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:29:23.782905    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b8a3de_cc6b_489f_8a1d_8be74509e423.slice/crio-1687a5426584d2b4831dfe40a87a547bc126e80c607185320632a697e1106ca4 WatchSource:0}: Error finding container 1687a5426584d2b4831dfe40a87a547bc126e80c607185320632a697e1106ca4: Status 404 returned error can't find the container with id 1687a5426584d2b4831dfe40a87a547bc126e80c607185320632a697e1106ca4
Nov 03 01:29:23.992782 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:23.992750    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d68kq" event={"ID":"c8b8a3de-cc6b-489f-8a1d-8be74509e423","Type":"ContainerStarted","Data":"1687a5426584d2b4831dfe40a87a547bc126e80c607185320632a697e1106ca4"}
Nov 03 01:29:25.004198 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:25.004096    5012 generic.go:356] "Generic (PLEG): container finished" podID="c8b8a3de-cc6b-489f-8a1d-8be74509e423" containerID="1c80bf1eb09ba9192af988785b6786d02d5a9f6d2439a13b73d31484e540abaa" exitCode=0
Nov 03 01:29:25.004703 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:25.004196    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d68kq" event={"ID":"c8b8a3de-cc6b-489f-8a1d-8be74509e423","Type":"ContainerDied","Data":"1c80bf1eb09ba9192af988785b6786d02d5a9f6d2439a13b73d31484e540abaa"}
Nov 03 01:29:26.374591 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:26.374564    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d68kq"
Nov 03 01:29:26.471396 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:26.471361    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8b8a3de-cc6b-489f-8a1d-8be74509e423-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b8a3de-cc6b-489f-8a1d-8be74509e423-maas-api-openshift-ca\") pod \"c8b8a3de-cc6b-489f-8a1d-8be74509e423\" (UID: \"c8b8a3de-cc6b-489f-8a1d-8be74509e423\") "
Nov 03 01:29:26.471589 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:26.471421    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8b8a3de-cc6b-489f-8a1d-8be74509e423-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b8a3de-cc6b-489f-8a1d-8be74509e423-maas-api-xks-ca\") pod \"c8b8a3de-cc6b-489f-8a1d-8be74509e423\" (UID: \"c8b8a3de-cc6b-489f-8a1d-8be74509e423\") "
Nov 03 01:29:26.471589 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:26.471466    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b8a3de-cc6b-489f-8a1d-8be74509e423-kube-api-access-mqwc5\" (UniqueName: \"kubernetes.io/projected/c8b8a3de-cc6b-489f-8a1d-8be74509e423-kube-api-access-mqwc5\") pod \"c8b8a3de-cc6b-489f-8a1d-8be74509e423\" (UID: \"c8b8a3de-cc6b-489f-8a1d-8be74509e423\") "
Nov 03 01:29:26.471811 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:26.471780    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b8a3de-cc6b-489f-8a1d-8be74509e423-maas-api-openshift-ca" pod "c8b8a3de-cc6b-489f-8a1d-8be74509e423" (UID: "c8b8a3de-cc6b-489f-8a1d-8be74509e423"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:29:26.471942 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:26.471900    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b8a3de-cc6b-489f-8a1d-8be74509e423-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:29:26.474481 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:26.474454    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8b8a3de-cc6b-489f-8a1d-8be74509e423-maas-api-xks-ca" pod "c8b8a3de-cc6b-489f-8a1d-8be74509e423" (UID: "c8b8a3de-cc6b-489f-8a1d-8be74509e423"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:29:26.474579 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:26.474543    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b8a3de-cc6b-489f-8a1d-8be74509e423-kube-api-access-mqwc5" pod "c8b8a3de-cc6b-489f-8a1d-8be74509e423" (UID: "c8b8a3de-cc6b-489f-8a1d-8be74509e423"). InnerVolumeSpecName "kube-api-access-mqwc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:29:26.572506 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:26.572447    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b8a3de-cc6b-489f-8a1d-8be74509e423-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:29:26.572506 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:26.572470    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqwc5\" (UniqueName: \"kubernetes.io/projected/c8b8a3de-cc6b-489f-8a1d-8be74509e423-kube-api-access-mqwc5\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:29:27.058177 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:27.058088    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d68kq" event={"ID":"c8b8a3de-cc6b-489f-8a1d-8be74509e423","Type":"ContainerDied","Data":"1687a5426584d2b4831dfe40a87a547bc126e80c607185320632a697e1106ca4"}
Nov 03 01:29:27.058177 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:27.058144    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d68kq"
Nov 03 01:29:27.058619 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:27.058155    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1687a5426584d2b4831dfe40a87a547bc126e80c607185320632a697e1106ca4"
Nov 03 01:29:37.362475 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:29:37.362431    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:29:49.481117 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:49.481081    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bzl6r"]
Nov 03 01:29:49.481540 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:49.481485    5012 cpu_manager.go:414] "containerMap: removing container" podUID="c8b8a3de-cc6b-489f-8a1d-8be74509e423" containerName="revoke-keys"
Nov 03 01:29:49.481540 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:49.481496    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b8a3de-cc6b-489f-8a1d-8be74509e423" containerName="revoke-keys"
Nov 03 01:29:49.481617 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:49.481597    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b8a3de-cc6b-489f-8a1d-8be74509e423" containerName="revoke-keys"
Nov 03 01:29:49.488290 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:49.488258    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bzl6r"
Nov 03 01:29:49.502277 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:49.502249    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bzl6r"]
Nov 03 01:29:49.506991 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:49.506963    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:29:49.507159 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:49.507000    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:29:49.508483 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:49.508461    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:29:49.508483 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:49.508462    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:29:49.584872 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:49.584837    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e70b630-2c9d-49fe-89a1-f1249d7941f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-bzl6r\" (UID: \"0e70b630-2c9d-49fe-89a1-f1249d7941f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bzl6r"
Nov 03 01:29:49.585048 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:49.584875    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9xhk\" (UniqueName: \"kubernetes.io/projected/0e70b630-2c9d-49fe-89a1-f1249d7941f7-kube-api-access-v9xhk\") pod \"maas-api-revoke-sub-ae594979-bzl6r\" (UID: \"0e70b630-2c9d-49fe-89a1-f1249d7941f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bzl6r"
Nov 03 01:29:49.585159 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:49.585100    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e70b630-2c9d-49fe-89a1-f1249d7941f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-bzl6r\" (UID: \"0e70b630-2c9d-49fe-89a1-f1249d7941f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bzl6r"
Nov 03 01:29:49.686310 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:49.686281    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e70b630-2c9d-49fe-89a1-f1249d7941f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-bzl6r\" (UID: \"0e70b630-2c9d-49fe-89a1-f1249d7941f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bzl6r"
Nov 03 01:29:49.686429 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:49.686330    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e70b630-2c9d-49fe-89a1-f1249d7941f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-bzl6r\" (UID: \"0e70b630-2c9d-49fe-89a1-f1249d7941f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bzl6r"
Nov 03 01:29:49.686506 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:49.686454    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9xhk\" (UniqueName: \"kubernetes.io/projected/0e70b630-2c9d-49fe-89a1-f1249d7941f7-kube-api-access-v9xhk\") pod \"maas-api-revoke-sub-ae594979-bzl6r\" (UID: \"0e70b630-2c9d-49fe-89a1-f1249d7941f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bzl6r"
Nov 03 01:29:49.687040 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:49.687022    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e70b630-2c9d-49fe-89a1-f1249d7941f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-bzl6r\" (UID: \"0e70b630-2c9d-49fe-89a1-f1249d7941f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bzl6r"
Nov 03 01:29:49.688583 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:49.688562    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e70b630-2c9d-49fe-89a1-f1249d7941f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-bzl6r\" (UID: \"0e70b630-2c9d-49fe-89a1-f1249d7941f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bzl6r"
Nov 03 01:29:49.703396 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:49.703376    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9xhk\" (UniqueName: \"kubernetes.io/projected/0e70b630-2c9d-49fe-89a1-f1249d7941f7-kube-api-access-v9xhk\") pod \"maas-api-revoke-sub-ae594979-bzl6r\" (UID: \"0e70b630-2c9d-49fe-89a1-f1249d7941f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bzl6r"
Nov 03 01:29:49.803721 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:49.803668    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bzl6r"
Nov 03 01:29:50.097214 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:50.097186    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bzl6r"]
Nov 03 01:29:50.097869 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:29:50.097843    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e70b630_2c9d_49fe_89a1_f1249d7941f7.slice/crio-f2a6300a0837ce08dbbc7820ea212e544fdceb574bdbf827e3924e47d7d8e509 WatchSource:0}: Error finding container f2a6300a0837ce08dbbc7820ea212e544fdceb574bdbf827e3924e47d7d8e509: Status 404 returned error can't find the container with id f2a6300a0837ce08dbbc7820ea212e544fdceb574bdbf827e3924e47d7d8e509
Nov 03 01:29:50.201678 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:50.201646    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bzl6r" event={"ID":"0e70b630-2c9d-49fe-89a1-f1249d7941f7","Type":"ContainerStarted","Data":"f2a6300a0837ce08dbbc7820ea212e544fdceb574bdbf827e3924e47d7d8e509"}
Nov 03 01:29:51.209144 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:51.209077    5012 generic.go:356] "Generic (PLEG): container finished" podID="0e70b630-2c9d-49fe-89a1-f1249d7941f7" containerID="dfba3335cf69a2ede98dd7e0b45eeb8d79bc232365e610378080e78b8deeed2b" exitCode=0
Nov 03 01:29:51.209459 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:51.209159    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bzl6r" event={"ID":"0e70b630-2c9d-49fe-89a1-f1249d7941f7","Type":"ContainerDied","Data":"dfba3335cf69a2ede98dd7e0b45eeb8d79bc232365e610378080e78b8deeed2b"}
Nov 03 01:29:52.375433 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:52.375404    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bzl6r"
Nov 03 01:29:52.511744 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:52.511662    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e70b630-2c9d-49fe-89a1-f1249d7941f7-kube-api-access-v9xhk\" (UniqueName: \"kubernetes.io/projected/0e70b630-2c9d-49fe-89a1-f1249d7941f7-kube-api-access-v9xhk\") pod \"0e70b630-2c9d-49fe-89a1-f1249d7941f7\" (UID: \"0e70b630-2c9d-49fe-89a1-f1249d7941f7\") "
Nov 03 01:29:52.511893 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:52.511766    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e70b630-2c9d-49fe-89a1-f1249d7941f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e70b630-2c9d-49fe-89a1-f1249d7941f7-maas-api-openshift-ca\") pod \"0e70b630-2c9d-49fe-89a1-f1249d7941f7\" (UID: \"0e70b630-2c9d-49fe-89a1-f1249d7941f7\") "
Nov 03 01:29:52.511893 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:52.511828    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e70b630-2c9d-49fe-89a1-f1249d7941f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e70b630-2c9d-49fe-89a1-f1249d7941f7-maas-api-xks-ca\") pod \"0e70b630-2c9d-49fe-89a1-f1249d7941f7\" (UID: \"0e70b630-2c9d-49fe-89a1-f1249d7941f7\") "
Nov 03 01:29:52.512132 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:52.512109    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e70b630-2c9d-49fe-89a1-f1249d7941f7-maas-api-openshift-ca" pod "0e70b630-2c9d-49fe-89a1-f1249d7941f7" (UID: "0e70b630-2c9d-49fe-89a1-f1249d7941f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:29:52.513737 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:52.513714    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e70b630-2c9d-49fe-89a1-f1249d7941f7-kube-api-access-v9xhk" pod "0e70b630-2c9d-49fe-89a1-f1249d7941f7" (UID: "0e70b630-2c9d-49fe-89a1-f1249d7941f7"). InnerVolumeSpecName "kube-api-access-v9xhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:29:52.513819 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:52.513757    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e70b630-2c9d-49fe-89a1-f1249d7941f7-maas-api-xks-ca" pod "0e70b630-2c9d-49fe-89a1-f1249d7941f7" (UID: "0e70b630-2c9d-49fe-89a1-f1249d7941f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:29:52.612535 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:52.612501    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e70b630-2c9d-49fe-89a1-f1249d7941f7-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:29:52.612535 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:52.612535    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e70b630-2c9d-49fe-89a1-f1249d7941f7-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:29:52.612535 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:52.612546    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9xhk\" (UniqueName: \"kubernetes.io/projected/0e70b630-2c9d-49fe-89a1-f1249d7941f7-kube-api-access-v9xhk\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:29:53.025636 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:53.025607    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b9n22"]
Nov 03 01:29:53.036041 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:53.036016    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b9n22"]
Nov 03 01:29:53.230153 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:53.230116    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bzl6r" event={"ID":"0e70b630-2c9d-49fe-89a1-f1249d7941f7","Type":"ContainerDied","Data":"f2a6300a0837ce08dbbc7820ea212e544fdceb574bdbf827e3924e47d7d8e509"}
Nov 03 01:29:53.230153 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:53.230139    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bzl6r"
Nov 03 01:29:53.230403 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:53.230194    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2a6300a0837ce08dbbc7820ea212e544fdceb574bdbf827e3924e47d7d8e509"
Nov 03 01:29:53.242241 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:29:53.242217    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bce404fd-5ded-4ffe-b555-459d81edab3c" path="/var/lib/kubelet/pods/bce404fd-5ded-4ffe-b555-459d81edab3c/volumes"
Nov 03 01:30:00.142287 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:00.142213    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7"]
Nov 03 01:30:00.142656 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:00.142622    5012 cpu_manager.go:414] "containerMap: removing container" podUID="0e70b630-2c9d-49fe-89a1-f1249d7941f7" containerName="revoke-keys"
Nov 03 01:30:00.142656 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:00.142634    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e70b630-2c9d-49fe-89a1-f1249d7941f7" containerName="revoke-keys"
Nov 03 01:30:00.142739 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:00.142718    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e70b630-2c9d-49fe-89a1-f1249d7941f7" containerName="revoke-keys"
Nov 03 01:30:00.147188 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:00.147169    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7"
Nov 03 01:30:00.164677 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:00.164648    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7"]
Nov 03 01:30:00.170838 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:00.170809    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:30:00.170966 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:00.170929    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:30:00.171092 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:00.171075    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:30:00.284705 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:00.284669    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbbc5\" (UniqueName: \"kubernetes.io/projected/1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6-kube-api-access-cbbc5\") pod \"maas-api-key-cleanup-29894490-hl7h7\" (UID: \"1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7"
Nov 03 01:30:00.284887 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:00.284848    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894490-hl7h7\" (UID: \"1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7"
Nov 03 01:30:00.385676 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:00.385636    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894490-hl7h7\" (UID: \"1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7"
Nov 03 01:30:00.385854 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:00.385690    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbbc5\" (UniqueName: \"kubernetes.io/projected/1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6-kube-api-access-cbbc5\") pod \"maas-api-key-cleanup-29894490-hl7h7\" (UID: \"1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7"
Nov 03 01:30:00.386354 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:00.386330    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894490-hl7h7\" (UID: \"1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7"
Nov 03 01:30:00.402702 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:00.402638    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbbc5\" (UniqueName: \"kubernetes.io/projected/1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6-kube-api-access-cbbc5\") pod \"maas-api-key-cleanup-29894490-hl7h7\" (UID: \"1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7"
Nov 03 01:30:00.460344 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:00.460317    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7"
Nov 03 01:30:00.754204 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:00.754178    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7"]
Nov 03 01:30:00.755034 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:30:00.755008    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1645d6aa_5c7a_461e_9e40_4fd31fbb5aa6.slice/crio-3494db81790c2f70a7ac7441035d00f2d1ba31f2ba704a566bb4104e0b1d6864 WatchSource:0}: Error finding container 3494db81790c2f70a7ac7441035d00f2d1ba31f2ba704a566bb4104e0b1d6864: Status 404 returned error can't find the container with id 3494db81790c2f70a7ac7441035d00f2d1ba31f2ba704a566bb4104e0b1d6864
Nov 03 01:30:01.279238 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:01.279198    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7" event={"ID":"1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6","Type":"ContainerStarted","Data":"a2c086de1355dcac44483e12f29abfce3e8f4fde2613e57bd6dbb0731edc19f9"}
Nov 03 01:30:01.279643 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:01.279252    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7" event={"ID":"1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6","Type":"ContainerStarted","Data":"3494db81790c2f70a7ac7441035d00f2d1ba31f2ba704a566bb4104e0b1d6864"}
Nov 03 01:30:01.314801 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:01.314757    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7" podStartSLOduration=1.314744373 podStartE2EDuration="1.314744373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:01.312017712 +0000 UTC m=+5084.191601755" watchObservedRunningTime="2026-11-03 01:30:01.314744373 +0000 UTC m=+5084.194328441"
Nov 03 01:30:02.287344 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:02.287307    5012 generic.go:356] "Generic (PLEG): container finished" podID="1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6" containerID="a2c086de1355dcac44483e12f29abfce3e8f4fde2613e57bd6dbb0731edc19f9" exitCode=60
Nov 03 01:30:02.287802 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:02.287386    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7" event={"ID":"1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6","Type":"ContainerDied","Data":"a2c086de1355dcac44483e12f29abfce3e8f4fde2613e57bd6dbb0731edc19f9"}
Nov 03 01:30:02.287802 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:02.287600    5012 scope.go:122] "RemoveContainer" containerID="a2c086de1355dcac44483e12f29abfce3e8f4fde2613e57bd6dbb0731edc19f9"
Nov 03 01:30:03.294572 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:03.294494    5012 generic.go:356] "Generic (PLEG): container finished" podID="1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6" containerID="9d9cc4c213430da951418ba19e4bf61db0ddaca6ce63560aab7ad881546c020b" exitCode=60
Nov 03 01:30:03.295008 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:03.294582    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7" event={"ID":"1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6","Type":"ContainerDied","Data":"9d9cc4c213430da951418ba19e4bf61db0ddaca6ce63560aab7ad881546c020b"}
Nov 03 01:30:03.295008 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:03.294637    5012 scope.go:122] "RemoveContainer" containerID="a2c086de1355dcac44483e12f29abfce3e8f4fde2613e57bd6dbb0731edc19f9"
Nov 03 01:30:03.295008 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:03.294751    5012 scope.go:122] "RemoveContainer" containerID="9d9cc4c213430da951418ba19e4bf61db0ddaca6ce63560aab7ad881546c020b"
Nov 03 01:30:03.295008 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:30:03.294997    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894490-hl7h7_odh-ai-gateway-infra(1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7" podUID="1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6"
Nov 03 01:30:04.303171 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:04.303137    5012 scope.go:122] "RemoveContainer" containerID="9d9cc4c213430da951418ba19e4bf61db0ddaca6ce63560aab7ad881546c020b"
Nov 03 01:30:04.303611 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:30:04.303378    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894490-hl7h7_odh-ai-gateway-infra(1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7" podUID="1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6"
Nov 03 01:30:07.362045 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:30:07.362008    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:30:10.032162 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:10.032124    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rhttb"]
Nov 03 01:30:10.042050 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:10.042014    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rhttb"]
Nov 03 01:30:11.243450 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:11.243411    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d69a1ce9-cd85-4f0d-b6a7-5317dc21c645" path="/var/lib/kubelet/pods/d69a1ce9-cd85-4f0d-b6a7-5317dc21c645/volumes"
Nov 03 01:30:15.236184 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:15.236151    5012 scope.go:122] "RemoveContainer" containerID="9d9cc4c213430da951418ba19e4bf61db0ddaca6ce63560aab7ad881546c020b"
Nov 03 01:30:16.384867 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:16.384834    5012 generic.go:356] "Generic (PLEG): container finished" podID="1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6" containerID="a1791dd32dd0119de8fe2f38eda2b3dde90e2771fde39a47712fd9f0bca459f2" exitCode=60
Nov 03 01:30:16.385332 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:16.384900    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7" event={"ID":"1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6","Type":"ContainerDied","Data":"a1791dd32dd0119de8fe2f38eda2b3dde90e2771fde39a47712fd9f0bca459f2"}
Nov 03 01:30:16.385332 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:16.384981    5012 scope.go:122] "RemoveContainer" containerID="9d9cc4c213430da951418ba19e4bf61db0ddaca6ce63560aab7ad881546c020b"
Nov 03 01:30:16.385332 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:16.385265    5012 scope.go:122] "RemoveContainer" containerID="a1791dd32dd0119de8fe2f38eda2b3dde90e2771fde39a47712fd9f0bca459f2"
Nov 03 01:30:16.385511 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:30:16.385465    5012 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894490-hl7h7_odh-ai-gateway-infra(1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7" podUID="1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6"
Nov 03 01:30:17.432037 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:17.432005    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7"]
Nov 03 01:30:17.569126 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:17.569100    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7"
Nov 03 01:30:17.589970 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:17.589942    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6-maas-api-service-ca\") pod \"1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6\" (UID: \"1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6\") "
Nov 03 01:30:17.590076 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:17.589977    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6-kube-api-access-cbbc5\" (UniqueName: \"kubernetes.io/projected/1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6-kube-api-access-cbbc5\") pod \"1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6\" (UID: \"1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6\") "
Nov 03 01:30:17.590339 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:17.590313    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6-maas-api-service-ca" pod "1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6" (UID: "1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:17.591895 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:17.591857    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6-kube-api-access-cbbc5" pod "1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6" (UID: "1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6"). InnerVolumeSpecName "kube-api-access-cbbc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:17.691360 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:17.691266    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6-maas-api-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:30:17.691360 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:17.691302    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbbc5\" (UniqueName: \"kubernetes.io/projected/1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6-kube-api-access-cbbc5\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:30:18.405603 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:18.405570    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7"
Nov 03 01:30:18.405804 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:18.405607    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7" event={"ID":"1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6","Type":"ContainerDied","Data":"3494db81790c2f70a7ac7441035d00f2d1ba31f2ba704a566bb4104e0b1d6864"}
Nov 03 01:30:18.405804 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:18.405681    5012 scope.go:122] "RemoveContainer" containerID="a1791dd32dd0119de8fe2f38eda2b3dde90e2771fde39a47712fd9f0bca459f2"
Nov 03 01:30:18.451888 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:18.451860    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7"]
Nov 03 01:30:18.463073 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:18.463049    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894490-hl7h7"]
Nov 03 01:30:18.911543 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:18.911513    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 01:30:18.916151 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:18.916128    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 01:30:18.917584 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:18.917563    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 01:30:18.922537 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:18.922519    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 01:30:19.242988 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:19.242900    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6" path="/var/lib/kubelet/pods/1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6/volumes"
Nov 03 01:30:21.775049 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:21.775004    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-prg8l"]
Nov 03 01:30:21.775691 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:21.775667    5012 cpu_manager.go:414] "containerMap: removing container" podUID="1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6" containerName="cleanup"
Nov 03 01:30:21.775746 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:21.775696    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6" containerName="cleanup"
Nov 03 01:30:21.775746 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:21.775710    5012 cpu_manager.go:414] "containerMap: removing container" podUID="1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6" containerName="cleanup"
Nov 03 01:30:21.775746 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:21.775719    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6" containerName="cleanup"
Nov 03 01:30:21.775888 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:21.775874    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6" containerName="cleanup"
Nov 03 01:30:21.775956 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:21.775897    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6" containerName="cleanup"
Nov 03 01:30:21.780790 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:21.780772    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-prg8l"
Nov 03 01:30:21.802401 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:21.800126    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-prg8l"]
Nov 03 01:30:21.806932 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:21.805190    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:30:21.806932 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:21.805304    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:30:21.806932 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:21.805356    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:30:21.813063 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:21.813040    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:30:21.853479 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:21.853447    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg9sl\" (UniqueName: \"kubernetes.io/projected/175f0a30-66ed-4f93-b5f9-32b956dfc6f0-kube-api-access-mg9sl\") pod \"maas-api-revoke-sub-79805ca9-prg8l\" (UID: \"175f0a30-66ed-4f93-b5f9-32b956dfc6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-prg8l"
Nov 03 01:30:21.871425 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:21.871397    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175f0a30-66ed-4f93-b5f9-32b956dfc6f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-prg8l\" (UID: \"175f0a30-66ed-4f93-b5f9-32b956dfc6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-prg8l"
Nov 03 01:30:21.871631 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:21.871614    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175f0a30-66ed-4f93-b5f9-32b956dfc6f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-prg8l\" (UID: \"175f0a30-66ed-4f93-b5f9-32b956dfc6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-prg8l"
Nov 03 01:30:21.972451 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:21.972418    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg9sl\" (UniqueName: \"kubernetes.io/projected/175f0a30-66ed-4f93-b5f9-32b956dfc6f0-kube-api-access-mg9sl\") pod \"maas-api-revoke-sub-79805ca9-prg8l\" (UID: \"175f0a30-66ed-4f93-b5f9-32b956dfc6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-prg8l"
Nov 03 01:30:21.972603 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:21.972463    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175f0a30-66ed-4f93-b5f9-32b956dfc6f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-prg8l\" (UID: \"175f0a30-66ed-4f93-b5f9-32b956dfc6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-prg8l"
Nov 03 01:30:21.972603 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:21.972493    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175f0a30-66ed-4f93-b5f9-32b956dfc6f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-prg8l\" (UID: \"175f0a30-66ed-4f93-b5f9-32b956dfc6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-prg8l"
Nov 03 01:30:21.973243 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:21.973222    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175f0a30-66ed-4f93-b5f9-32b956dfc6f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-prg8l\" (UID: \"175f0a30-66ed-4f93-b5f9-32b956dfc6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-prg8l"
Nov 03 01:30:21.974976 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:21.974959    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175f0a30-66ed-4f93-b5f9-32b956dfc6f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-prg8l\" (UID: \"175f0a30-66ed-4f93-b5f9-32b956dfc6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-prg8l"
Nov 03 01:30:22.043671 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:22.043582    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg9sl\" (UniqueName: \"kubernetes.io/projected/175f0a30-66ed-4f93-b5f9-32b956dfc6f0-kube-api-access-mg9sl\") pod \"maas-api-revoke-sub-79805ca9-prg8l\" (UID: \"175f0a30-66ed-4f93-b5f9-32b956dfc6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-prg8l"
Nov 03 01:30:22.120932 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:22.120028    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-prg8l"
Nov 03 01:30:22.557025 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:22.556989    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-prg8l"]
Nov 03 01:30:23.472616 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:23.472580    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-prg8l" event={"ID":"175f0a30-66ed-4f93-b5f9-32b956dfc6f0","Type":"ContainerStarted","Data":"ff1742cd52547c37b07be04acf611cd997e3f9b3818f5dcd73d0897bd18cc2b9"}
Nov 03 01:30:23.473059 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:23.472634    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-prg8l" event={"ID":"175f0a30-66ed-4f93-b5f9-32b956dfc6f0","Type":"ContainerStarted","Data":"809950ae42502cbfda60d25a51ff13881cf27caba917193321385bbdee52efe6"}
Nov 03 01:30:23.509114 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:23.509065    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-prg8l" podStartSLOduration=2.509047973 podStartE2EDuration="2.509047973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:23.504753225 +0000 UTC m=+5106.384337268" watchObservedRunningTime="2026-11-03 01:30:23.509047973 +0000 UTC m=+5106.388632016"
Nov 03 01:30:24.479790 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:24.479753    5012 generic.go:356] "Generic (PLEG): container finished" podID="175f0a30-66ed-4f93-b5f9-32b956dfc6f0" containerID="ff1742cd52547c37b07be04acf611cd997e3f9b3818f5dcd73d0897bd18cc2b9" exitCode=0
Nov 03 01:30:24.480222 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:24.479832    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-prg8l" event={"ID":"175f0a30-66ed-4f93-b5f9-32b956dfc6f0","Type":"ContainerDied","Data":"ff1742cd52547c37b07be04acf611cd997e3f9b3818f5dcd73d0897bd18cc2b9"}
Nov 03 01:30:25.624679 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:25.624650    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-prg8l"
Nov 03 01:30:25.720430 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:25.720398    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/175f0a30-66ed-4f93-b5f9-32b956dfc6f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175f0a30-66ed-4f93-b5f9-32b956dfc6f0-maas-api-openshift-ca\") pod \"175f0a30-66ed-4f93-b5f9-32b956dfc6f0\" (UID: \"175f0a30-66ed-4f93-b5f9-32b956dfc6f0\") "
Nov 03 01:30:25.720591 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:25.720441    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/175f0a30-66ed-4f93-b5f9-32b956dfc6f0-kube-api-access-mg9sl\" (UniqueName: \"kubernetes.io/projected/175f0a30-66ed-4f93-b5f9-32b956dfc6f0-kube-api-access-mg9sl\") pod \"175f0a30-66ed-4f93-b5f9-32b956dfc6f0\" (UID: \"175f0a30-66ed-4f93-b5f9-32b956dfc6f0\") "
Nov 03 01:30:25.720591 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:25.720519    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/175f0a30-66ed-4f93-b5f9-32b956dfc6f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175f0a30-66ed-4f93-b5f9-32b956dfc6f0-maas-api-xks-ca\") pod \"175f0a30-66ed-4f93-b5f9-32b956dfc6f0\" (UID: \"175f0a30-66ed-4f93-b5f9-32b956dfc6f0\") "
Nov 03 01:30:25.720816 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:25.720783    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/175f0a30-66ed-4f93-b5f9-32b956dfc6f0-maas-api-openshift-ca" pod "175f0a30-66ed-4f93-b5f9-32b956dfc6f0" (UID: "175f0a30-66ed-4f93-b5f9-32b956dfc6f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:25.720880 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:25.720867    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175f0a30-66ed-4f93-b5f9-32b956dfc6f0-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:30:25.722635 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:25.722603    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/175f0a30-66ed-4f93-b5f9-32b956dfc6f0-maas-api-xks-ca" pod "175f0a30-66ed-4f93-b5f9-32b956dfc6f0" (UID: "175f0a30-66ed-4f93-b5f9-32b956dfc6f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:30:25.722772 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:25.722644    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175f0a30-66ed-4f93-b5f9-32b956dfc6f0-kube-api-access-mg9sl" pod "175f0a30-66ed-4f93-b5f9-32b956dfc6f0" (UID: "175f0a30-66ed-4f93-b5f9-32b956dfc6f0"). InnerVolumeSpecName "kube-api-access-mg9sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:25.822277 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:25.822207    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg9sl\" (UniqueName: \"kubernetes.io/projected/175f0a30-66ed-4f93-b5f9-32b956dfc6f0-kube-api-access-mg9sl\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:30:25.822277 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:25.822239    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175f0a30-66ed-4f93-b5f9-32b956dfc6f0-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:30:26.494137 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:26.494110    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-prg8l"
Nov 03 01:30:26.494339 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:26.494142    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-prg8l" event={"ID":"175f0a30-66ed-4f93-b5f9-32b956dfc6f0","Type":"ContainerDied","Data":"809950ae42502cbfda60d25a51ff13881cf27caba917193321385bbdee52efe6"}
Nov 03 01:30:26.494339 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:26.494192    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="809950ae42502cbfda60d25a51ff13881cf27caba917193321385bbdee52efe6"
Nov 03 01:30:30.026882 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:30.026845    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ef9843-rtcwf"]
Nov 03 01:30:30.127537 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:30.127503    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ef9843-rtcwf"]
Nov 03 01:30:31.242074 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:31.242038    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bff2a9d-336f-43b2-ba62-14d7bca26a9a" path="/var/lib/kubelet/pods/0bff2a9d-336f-43b2-ba62-14d7bca26a9a/volumes"
Nov 03 01:30:35.555638 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:35.555604    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2sbm"]
Nov 03 01:30:35.556838 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:35.556811    5012 cpu_manager.go:414] "containerMap: removing container" podUID="175f0a30-66ed-4f93-b5f9-32b956dfc6f0" containerName="revoke-keys"
Nov 03 01:30:35.557014 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:35.556994    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="175f0a30-66ed-4f93-b5f9-32b956dfc6f0" containerName="revoke-keys"
Nov 03 01:30:35.557123 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:35.557109    5012 cpu_manager.go:414] "containerMap: removing container" podUID="1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6" containerName="cleanup"
Nov 03 01:30:35.557227 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:35.557213    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6" containerName="cleanup"
Nov 03 01:30:35.557477 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:35.557459    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="175f0a30-66ed-4f93-b5f9-32b956dfc6f0" containerName="revoke-keys"
Nov 03 01:30:35.557592 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:35.557580    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="1645d6aa-5c7a-461e-9e40-4fd31fbb5aa6" containerName="cleanup"
Nov 03 01:30:35.567517 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:35.566186    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2sbm"
Nov 03 01:30:35.571726 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:35.571699    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2sbm"]
Nov 03 01:30:35.579982 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:35.579957    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:30:35.598178 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:35.598155    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:30:35.598419 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:35.598399    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:30:35.598621 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:35.598603    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:30:35.742933 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:35.742486    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03873333-551f-4b0d-9ccd-fa856943acf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q2sbm\" (UID: \"03873333-551f-4b0d-9ccd-fa856943acf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2sbm"
Nov 03 01:30:35.742933 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:35.742582    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxq2z\" (UniqueName: \"kubernetes.io/projected/03873333-551f-4b0d-9ccd-fa856943acf0-kube-api-access-cxq2z\") pod \"maas-api-revoke-sub-0b7e8edb-q2sbm\" (UID: \"03873333-551f-4b0d-9ccd-fa856943acf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2sbm"
Nov 03 01:30:35.742933 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:35.742702    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03873333-551f-4b0d-9ccd-fa856943acf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q2sbm\" (UID: \"03873333-551f-4b0d-9ccd-fa856943acf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2sbm"
Nov 03 01:30:35.844568 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:35.844474    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03873333-551f-4b0d-9ccd-fa856943acf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q2sbm\" (UID: \"03873333-551f-4b0d-9ccd-fa856943acf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2sbm"
Nov 03 01:30:35.844842 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:35.844816    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03873333-551f-4b0d-9ccd-fa856943acf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q2sbm\" (UID: \"03873333-551f-4b0d-9ccd-fa856943acf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2sbm"
Nov 03 01:30:35.845042 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:35.845019    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxq2z\" (UniqueName: \"kubernetes.io/projected/03873333-551f-4b0d-9ccd-fa856943acf0-kube-api-access-cxq2z\") pod \"maas-api-revoke-sub-0b7e8edb-q2sbm\" (UID: \"03873333-551f-4b0d-9ccd-fa856943acf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2sbm"
Nov 03 01:30:35.846264 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:35.846241    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03873333-551f-4b0d-9ccd-fa856943acf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q2sbm\" (UID: \"03873333-551f-4b0d-9ccd-fa856943acf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2sbm"
Nov 03 01:30:35.865534 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:35.865511    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03873333-551f-4b0d-9ccd-fa856943acf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q2sbm\" (UID: \"03873333-551f-4b0d-9ccd-fa856943acf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2sbm"
Nov 03 01:30:35.869853 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:35.869826    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxq2z\" (UniqueName: \"kubernetes.io/projected/03873333-551f-4b0d-9ccd-fa856943acf0-kube-api-access-cxq2z\") pod \"maas-api-revoke-sub-0b7e8edb-q2sbm\" (UID: \"03873333-551f-4b0d-9ccd-fa856943acf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2sbm"
Nov 03 01:30:35.884028 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:35.884002    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2sbm"
Nov 03 01:30:36.355052 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:36.355016    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2sbm"]
Nov 03 01:30:36.571374 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:36.571341    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2sbm" event={"ID":"03873333-551f-4b0d-9ccd-fa856943acf0","Type":"ContainerStarted","Data":"006adf08bb2bdb01a4dc8b59f03a24b11d208b48985300df20444fc95cd57571"}
Nov 03 01:30:37.364154 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:30:37.364117    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:30:37.579490 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:37.579456    5012 generic.go:356] "Generic (PLEG): container finished" podID="03873333-551f-4b0d-9ccd-fa856943acf0" containerID="3b939bf914743854c71b0df0d5bcabf93168c3f8ce91dcfd1a20e91e0739ac2c" exitCode=0
Nov 03 01:30:37.579839 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:37.579505    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2sbm" event={"ID":"03873333-551f-4b0d-9ccd-fa856943acf0","Type":"ContainerDied","Data":"3b939bf914743854c71b0df0d5bcabf93168c3f8ce91dcfd1a20e91e0739ac2c"}
Nov 03 01:30:38.726317 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:38.726290    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2sbm"
Nov 03 01:30:38.789323 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:38.789294    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03873333-551f-4b0d-9ccd-fa856943acf0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03873333-551f-4b0d-9ccd-fa856943acf0-maas-api-openshift-ca\") pod \"03873333-551f-4b0d-9ccd-fa856943acf0\" (UID: \"03873333-551f-4b0d-9ccd-fa856943acf0\") "
Nov 03 01:30:38.789471 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:38.789336    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03873333-551f-4b0d-9ccd-fa856943acf0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03873333-551f-4b0d-9ccd-fa856943acf0-maas-api-xks-ca\") pod \"03873333-551f-4b0d-9ccd-fa856943acf0\" (UID: \"03873333-551f-4b0d-9ccd-fa856943acf0\") "
Nov 03 01:30:38.789471 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:38.789407    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03873333-551f-4b0d-9ccd-fa856943acf0-kube-api-access-cxq2z\" (UniqueName: \"kubernetes.io/projected/03873333-551f-4b0d-9ccd-fa856943acf0-kube-api-access-cxq2z\") pod \"03873333-551f-4b0d-9ccd-fa856943acf0\" (UID: \"03873333-551f-4b0d-9ccd-fa856943acf0\") "
Nov 03 01:30:38.789748 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:38.789719    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03873333-551f-4b0d-9ccd-fa856943acf0-maas-api-openshift-ca" pod "03873333-551f-4b0d-9ccd-fa856943acf0" (UID: "03873333-551f-4b0d-9ccd-fa856943acf0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:38.791555 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:38.791529    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03873333-551f-4b0d-9ccd-fa856943acf0-maas-api-xks-ca" pod "03873333-551f-4b0d-9ccd-fa856943acf0" (UID: "03873333-551f-4b0d-9ccd-fa856943acf0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:30:38.791669 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:38.791534    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03873333-551f-4b0d-9ccd-fa856943acf0-kube-api-access-cxq2z" pod "03873333-551f-4b0d-9ccd-fa856943acf0" (UID: "03873333-551f-4b0d-9ccd-fa856943acf0"). InnerVolumeSpecName "kube-api-access-cxq2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:38.890190 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:38.890156    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03873333-551f-4b0d-9ccd-fa856943acf0-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:30:38.890190 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:38.890189    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03873333-551f-4b0d-9ccd-fa856943acf0-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:30:38.890337 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:38.890205    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxq2z\" (UniqueName: \"kubernetes.io/projected/03873333-551f-4b0d-9ccd-fa856943acf0-kube-api-access-cxq2z\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:30:39.593418 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:39.593320    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2sbm" event={"ID":"03873333-551f-4b0d-9ccd-fa856943acf0","Type":"ContainerDied","Data":"006adf08bb2bdb01a4dc8b59f03a24b11d208b48985300df20444fc95cd57571"}
Nov 03 01:30:39.593418 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:39.593381    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="006adf08bb2bdb01a4dc8b59f03a24b11d208b48985300df20444fc95cd57571"
Nov 03 01:30:39.593418 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:39.593334    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2sbm"
Nov 03 01:30:46.288989 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:46.288950    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkxn2"]
Nov 03 01:30:46.289397 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:46.289346    5012 cpu_manager.go:414] "containerMap: removing container" podUID="03873333-551f-4b0d-9ccd-fa856943acf0" containerName="revoke-keys"
Nov 03 01:30:46.289397 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:46.289357    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03873333-551f-4b0d-9ccd-fa856943acf0" containerName="revoke-keys"
Nov 03 01:30:46.289487 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:46.289448    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="03873333-551f-4b0d-9ccd-fa856943acf0" containerName="revoke-keys"
Nov 03 01:30:46.292933 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:46.292885    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkxn2"
Nov 03 01:30:46.316856 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:46.316824    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:30:46.317027 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:46.316881    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:30:46.317027 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:46.316972    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:30:46.317027 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:46.317001    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:30:46.321516 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:46.321490    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkxn2"]
Nov 03 01:30:46.357339 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:46.357309    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4825f293-432b-4f3c-8518-a9475ef58025-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fkxn2\" (UID: \"4825f293-432b-4f3c-8518-a9475ef58025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkxn2"
Nov 03 01:30:46.357486 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:46.357462    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4825f293-432b-4f3c-8518-a9475ef58025-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fkxn2\" (UID: \"4825f293-432b-4f3c-8518-a9475ef58025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkxn2"
Nov 03 01:30:46.357633 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:46.357613    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2pj6\" (UniqueName: \"kubernetes.io/projected/4825f293-432b-4f3c-8518-a9475ef58025-kube-api-access-j2pj6\") pod \"maas-api-revoke-sub-3d4156f1-fkxn2\" (UID: \"4825f293-432b-4f3c-8518-a9475ef58025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkxn2"
Nov 03 01:30:46.458808 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:46.458776    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2pj6\" (UniqueName: \"kubernetes.io/projected/4825f293-432b-4f3c-8518-a9475ef58025-kube-api-access-j2pj6\") pod \"maas-api-revoke-sub-3d4156f1-fkxn2\" (UID: \"4825f293-432b-4f3c-8518-a9475ef58025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkxn2"
Nov 03 01:30:46.458947 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:46.458856    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4825f293-432b-4f3c-8518-a9475ef58025-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fkxn2\" (UID: \"4825f293-432b-4f3c-8518-a9475ef58025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkxn2"
Nov 03 01:30:46.458947 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:46.458935    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4825f293-432b-4f3c-8518-a9475ef58025-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fkxn2\" (UID: \"4825f293-432b-4f3c-8518-a9475ef58025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkxn2"
Nov 03 01:30:46.459510 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:46.459490    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4825f293-432b-4f3c-8518-a9475ef58025-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fkxn2\" (UID: \"4825f293-432b-4f3c-8518-a9475ef58025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkxn2"
Nov 03 01:30:46.461192 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:46.461171    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4825f293-432b-4f3c-8518-a9475ef58025-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fkxn2\" (UID: \"4825f293-432b-4f3c-8518-a9475ef58025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkxn2"
Nov 03 01:30:46.476449 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:46.476422    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2pj6\" (UniqueName: \"kubernetes.io/projected/4825f293-432b-4f3c-8518-a9475ef58025-kube-api-access-j2pj6\") pod \"maas-api-revoke-sub-3d4156f1-fkxn2\" (UID: \"4825f293-432b-4f3c-8518-a9475ef58025\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkxn2"
Nov 03 01:30:46.606840 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:46.606816    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkxn2"
Nov 03 01:30:46.898132 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:46.898102    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkxn2"]
Nov 03 01:30:46.899302 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:30:46.899273    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4825f293_432b_4f3c_8518_a9475ef58025.slice/crio-59b3f5b37e43e5510481229ffef48e7edd6ff93048b5ef8fe59704154cf0d197 WatchSource:0}: Error finding container 59b3f5b37e43e5510481229ffef48e7edd6ff93048b5ef8fe59704154cf0d197: Status 404 returned error can't find the container with id 59b3f5b37e43e5510481229ffef48e7edd6ff93048b5ef8fe59704154cf0d197
Nov 03 01:30:47.643460 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:47.643418    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkxn2" event={"ID":"4825f293-432b-4f3c-8518-a9475ef58025","Type":"ContainerStarted","Data":"4fb8ef7be238647aeb85b67e990b07d243a66b2293c67952a2713d860c90d036"}
Nov 03 01:30:47.643831 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:47.643485    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkxn2" event={"ID":"4825f293-432b-4f3c-8518-a9475ef58025","Type":"ContainerStarted","Data":"59b3f5b37e43e5510481229ffef48e7edd6ff93048b5ef8fe59704154cf0d197"}
Nov 03 01:30:47.677513 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:47.677464    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkxn2" podStartSLOduration=1.677452897 podStartE2EDuration="1.677452897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:47.676808469 +0000 UTC m=+5130.556392512" watchObservedRunningTime="2026-11-03 01:30:47.677452897 +0000 UTC m=+5130.557036940"
Nov 03 01:30:48.651543 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:48.651506    5012 generic.go:356] "Generic (PLEG): container finished" podID="4825f293-432b-4f3c-8518-a9475ef58025" containerID="4fb8ef7be238647aeb85b67e990b07d243a66b2293c67952a2713d860c90d036" exitCode=0
Nov 03 01:30:48.651989 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:48.651595    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkxn2" event={"ID":"4825f293-432b-4f3c-8518-a9475ef58025","Type":"ContainerDied","Data":"4fb8ef7be238647aeb85b67e990b07d243a66b2293c67952a2713d860c90d036"}
Nov 03 01:30:49.795815 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:49.795791    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkxn2"
Nov 03 01:30:49.889099 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:49.889067    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4825f293-432b-4f3c-8518-a9475ef58025-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4825f293-432b-4f3c-8518-a9475ef58025-maas-api-xks-ca\") pod \"4825f293-432b-4f3c-8518-a9475ef58025\" (UID: \"4825f293-432b-4f3c-8518-a9475ef58025\") "
Nov 03 01:30:49.889265 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:49.889106    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4825f293-432b-4f3c-8518-a9475ef58025-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4825f293-432b-4f3c-8518-a9475ef58025-maas-api-openshift-ca\") pod \"4825f293-432b-4f3c-8518-a9475ef58025\" (UID: \"4825f293-432b-4f3c-8518-a9475ef58025\") "
Nov 03 01:30:49.889309 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:49.889268    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4825f293-432b-4f3c-8518-a9475ef58025-kube-api-access-j2pj6\" (UniqueName: \"kubernetes.io/projected/4825f293-432b-4f3c-8518-a9475ef58025-kube-api-access-j2pj6\") pod \"4825f293-432b-4f3c-8518-a9475ef58025\" (UID: \"4825f293-432b-4f3c-8518-a9475ef58025\") "
Nov 03 01:30:49.889514 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:49.889491    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4825f293-432b-4f3c-8518-a9475ef58025-maas-api-openshift-ca" pod "4825f293-432b-4f3c-8518-a9475ef58025" (UID: "4825f293-432b-4f3c-8518-a9475ef58025"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:49.889631 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:49.889614    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4825f293-432b-4f3c-8518-a9475ef58025-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:30:49.891222 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:49.891185    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4825f293-432b-4f3c-8518-a9475ef58025-maas-api-xks-ca" pod "4825f293-432b-4f3c-8518-a9475ef58025" (UID: "4825f293-432b-4f3c-8518-a9475ef58025"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:30:49.891373 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:49.891273    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4825f293-432b-4f3c-8518-a9475ef58025-kube-api-access-j2pj6" pod "4825f293-432b-4f3c-8518-a9475ef58025" (UID: "4825f293-432b-4f3c-8518-a9475ef58025"). InnerVolumeSpecName "kube-api-access-j2pj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:49.990749 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:49.990692    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4825f293-432b-4f3c-8518-a9475ef58025-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:30:49.990749 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:49.990717    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2pj6\" (UniqueName: \"kubernetes.io/projected/4825f293-432b-4f3c-8518-a9475ef58025-kube-api-access-j2pj6\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:30:50.664860 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:50.664828    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkxn2"
Nov 03 01:30:50.665086 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:50.664853    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkxn2" event={"ID":"4825f293-432b-4f3c-8518-a9475ef58025","Type":"ContainerDied","Data":"59b3f5b37e43e5510481229ffef48e7edd6ff93048b5ef8fe59704154cf0d197"}
Nov 03 01:30:50.665086 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:30:50.664943    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59b3f5b37e43e5510481229ffef48e7edd6ff93048b5ef8fe59704154cf0d197"
Nov 03 01:31:03.502109 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:03.502075    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkxn2"]
Nov 03 01:31:03.512234 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:03.512211    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fkxn2"]
Nov 03 01:31:03.544691 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:03.544662    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nptmb"]
Nov 03 01:31:03.545271 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:03.545245    5012 cpu_manager.go:414] "containerMap: removing container" podUID="4825f293-432b-4f3c-8518-a9475ef58025" containerName="revoke-keys"
Nov 03 01:31:03.545271 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:03.545271    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4825f293-432b-4f3c-8518-a9475ef58025" containerName="revoke-keys"
Nov 03 01:31:03.545462 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:03.545415    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="4825f293-432b-4f3c-8518-a9475ef58025" containerName="revoke-keys"
Nov 03 01:31:03.548764 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:03.548746    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nptmb"
Nov 03 01:31:03.563339 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:03.563312    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:31:03.563951 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:03.563891    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:31:03.564052 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:03.564001    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:31:03.564052 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:03.564030    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:31:03.567451 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:03.567430    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nptmb"]
Nov 03 01:31:03.602236 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:03.602211    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e27b88ce-be07-4e3e-b127-c5bc29f1cb9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nptmb\" (UID: \"e27b88ce-be07-4e3e-b127-c5bc29f1cb9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nptmb"
Nov 03 01:31:03.602358 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:03.602243    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzxcx\" (UniqueName: \"kubernetes.io/projected/e27b88ce-be07-4e3e-b127-c5bc29f1cb9e-kube-api-access-mzxcx\") pod \"maas-api-revoke-sub-3d4156f1-nptmb\" (UID: \"e27b88ce-be07-4e3e-b127-c5bc29f1cb9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nptmb"
Nov 03 01:31:03.602457 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:03.602430    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e27b88ce-be07-4e3e-b127-c5bc29f1cb9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nptmb\" (UID: \"e27b88ce-be07-4e3e-b127-c5bc29f1cb9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nptmb"
Nov 03 01:31:03.703327 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:03.703297    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzxcx\" (UniqueName: \"kubernetes.io/projected/e27b88ce-be07-4e3e-b127-c5bc29f1cb9e-kube-api-access-mzxcx\") pod \"maas-api-revoke-sub-3d4156f1-nptmb\" (UID: \"e27b88ce-be07-4e3e-b127-c5bc29f1cb9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nptmb"
Nov 03 01:31:03.703454 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:03.703442    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e27b88ce-be07-4e3e-b127-c5bc29f1cb9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nptmb\" (UID: \"e27b88ce-be07-4e3e-b127-c5bc29f1cb9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nptmb"
Nov 03 01:31:03.703513 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:03.703497    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e27b88ce-be07-4e3e-b127-c5bc29f1cb9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nptmb\" (UID: \"e27b88ce-be07-4e3e-b127-c5bc29f1cb9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nptmb"
Nov 03 01:31:03.704131 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:03.704099    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e27b88ce-be07-4e3e-b127-c5bc29f1cb9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nptmb\" (UID: \"e27b88ce-be07-4e3e-b127-c5bc29f1cb9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nptmb"
Nov 03 01:31:03.705768 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:03.705750    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e27b88ce-be07-4e3e-b127-c5bc29f1cb9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nptmb\" (UID: \"e27b88ce-be07-4e3e-b127-c5bc29f1cb9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nptmb"
Nov 03 01:31:03.720165 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:03.720143    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzxcx\" (UniqueName: \"kubernetes.io/projected/e27b88ce-be07-4e3e-b127-c5bc29f1cb9e-kube-api-access-mzxcx\") pod \"maas-api-revoke-sub-3d4156f1-nptmb\" (UID: \"e27b88ce-be07-4e3e-b127-c5bc29f1cb9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nptmb"
Nov 03 01:31:03.861313 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:03.861284    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nptmb"
Nov 03 01:31:04.161736 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:04.158181    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nptmb"]
Nov 03 01:31:04.772646 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:04.772564    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nptmb" event={"ID":"e27b88ce-be07-4e3e-b127-c5bc29f1cb9e","Type":"ContainerStarted","Data":"4ec8004b70b4f760d4e8f70fb2e4dd68da46926f332d12bbc69c1cc20e3fad54"}
Nov 03 01:31:04.772646 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:04.772634    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nptmb" event={"ID":"e27b88ce-be07-4e3e-b127-c5bc29f1cb9e","Type":"ContainerStarted","Data":"68d69b6c2304e8c3a27d85d69497841c6692253f7b1b34732d58650d735f731d"}
Nov 03 01:31:04.815681 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:04.815628    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nptmb" podStartSLOduration=1.8156131439999998 podStartE2EDuration="1.815613144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:31:04.805646845 +0000 UTC m=+5147.685230889" watchObservedRunningTime="2026-11-03 01:31:04.815613144 +0000 UTC m=+5147.695197187"
Nov 03 01:31:05.242883 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:05.242851    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4825f293-432b-4f3c-8518-a9475ef58025" path="/var/lib/kubelet/pods/4825f293-432b-4f3c-8518-a9475ef58025/volumes"
Nov 03 01:31:05.780768 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:05.780733    5012 generic.go:356] "Generic (PLEG): container finished" podID="e27b88ce-be07-4e3e-b127-c5bc29f1cb9e" containerID="4ec8004b70b4f760d4e8f70fb2e4dd68da46926f332d12bbc69c1cc20e3fad54" exitCode=0
Nov 03 01:31:05.781165 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:05.780802    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nptmb" event={"ID":"e27b88ce-be07-4e3e-b127-c5bc29f1cb9e","Type":"ContainerDied","Data":"4ec8004b70b4f760d4e8f70fb2e4dd68da46926f332d12bbc69c1cc20e3fad54"}
Nov 03 01:31:06.923253 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:06.923228    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nptmb"
Nov 03 01:31:07.034290 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:07.034251    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e27b88ce-be07-4e3e-b127-c5bc29f1cb9e-kube-api-access-mzxcx\" (UniqueName: \"kubernetes.io/projected/e27b88ce-be07-4e3e-b127-c5bc29f1cb9e-kube-api-access-mzxcx\") pod \"e27b88ce-be07-4e3e-b127-c5bc29f1cb9e\" (UID: \"e27b88ce-be07-4e3e-b127-c5bc29f1cb9e\") "
Nov 03 01:31:07.034486 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:07.034347    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e27b88ce-be07-4e3e-b127-c5bc29f1cb9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e27b88ce-be07-4e3e-b127-c5bc29f1cb9e-maas-api-xks-ca\") pod \"e27b88ce-be07-4e3e-b127-c5bc29f1cb9e\" (UID: \"e27b88ce-be07-4e3e-b127-c5bc29f1cb9e\") "
Nov 03 01:31:07.034486 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:07.034411    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e27b88ce-be07-4e3e-b127-c5bc29f1cb9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e27b88ce-be07-4e3e-b127-c5bc29f1cb9e-maas-api-openshift-ca\") pod \"e27b88ce-be07-4e3e-b127-c5bc29f1cb9e\" (UID: \"e27b88ce-be07-4e3e-b127-c5bc29f1cb9e\") "
Nov 03 01:31:07.034795 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:07.034770    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e27b88ce-be07-4e3e-b127-c5bc29f1cb9e-maas-api-openshift-ca" pod "e27b88ce-be07-4e3e-b127-c5bc29f1cb9e" (UID: "e27b88ce-be07-4e3e-b127-c5bc29f1cb9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:31:07.036355 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:07.036333    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27b88ce-be07-4e3e-b127-c5bc29f1cb9e-kube-api-access-mzxcx" pod "e27b88ce-be07-4e3e-b127-c5bc29f1cb9e" (UID: "e27b88ce-be07-4e3e-b127-c5bc29f1cb9e"). InnerVolumeSpecName "kube-api-access-mzxcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:31:07.036459 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:07.036396    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e27b88ce-be07-4e3e-b127-c5bc29f1cb9e-maas-api-xks-ca" pod "e27b88ce-be07-4e3e-b127-c5bc29f1cb9e" (UID: "e27b88ce-be07-4e3e-b127-c5bc29f1cb9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:31:07.135012 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:07.134986    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e27b88ce-be07-4e3e-b127-c5bc29f1cb9e-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:31:07.135012 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:07.135009    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzxcx\" (UniqueName: \"kubernetes.io/projected/e27b88ce-be07-4e3e-b127-c5bc29f1cb9e-kube-api-access-mzxcx\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:31:07.135154 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:07.135019    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e27b88ce-be07-4e3e-b127-c5bc29f1cb9e-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:31:07.361774 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:31:07.361740    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:31:07.803709 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:07.803632    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nptmb"
Nov 03 01:31:07.803857 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:07.803636    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nptmb" event={"ID":"e27b88ce-be07-4e3e-b127-c5bc29f1cb9e","Type":"ContainerDied","Data":"68d69b6c2304e8c3a27d85d69497841c6692253f7b1b34732d58650d735f731d"}
Nov 03 01:31:07.803857 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:07.803790    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68d69b6c2304e8c3a27d85d69497841c6692253f7b1b34732d58650d735f731d"
Nov 03 01:31:22.183954 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:22.183921    5012 scope.go:122] "RemoveContainer" containerID="bc485aca5adcfa30658e8275931416b280890add7759648f7d8a96b72a1355a3"
Nov 03 01:31:22.192999 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:22.192980    5012 scope.go:122] "RemoveContainer" containerID="86bda7a1a3db942ef9c4a6514c940cb531e17c8dad7f0e0a4743fd0c59d42c28"
Nov 03 01:31:33.270261 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:33.270175    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-77dld"]
Nov 03 01:31:33.270925 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:33.270596    5012 cpu_manager.go:414] "containerMap: removing container" podUID="e27b88ce-be07-4e3e-b127-c5bc29f1cb9e" containerName="revoke-keys"
Nov 03 01:31:33.270925 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:33.270608    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e27b88ce-be07-4e3e-b127-c5bc29f1cb9e" containerName="revoke-keys"
Nov 03 01:31:33.270925 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:33.270697    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="e27b88ce-be07-4e3e-b127-c5bc29f1cb9e" containerName="revoke-keys"
Nov 03 01:31:33.273953 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:33.273900    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-77dld"
Nov 03 01:31:33.292596 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:33.292573    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-77dld"]
Nov 03 01:31:33.298178 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:33.298152    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:31:33.300058 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:33.300034    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:31:33.300185 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:33.300103    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:31:33.300258 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:33.300111    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:31:33.301506 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:33.301487    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da4e286-2604-498d-91f1-8c88d350eb4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-77dld\" (UID: \"3da4e286-2604-498d-91f1-8c88d350eb4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-77dld"
Nov 03 01:31:33.301624 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:33.301575    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fngqw\" (UniqueName: \"kubernetes.io/projected/3da4e286-2604-498d-91f1-8c88d350eb4a-kube-api-access-fngqw\") pod \"maas-api-revoke-sub-e2fc9c5b-77dld\" (UID: \"3da4e286-2604-498d-91f1-8c88d350eb4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-77dld"
Nov 03 01:31:33.301786 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:33.301759    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da4e286-2604-498d-91f1-8c88d350eb4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-77dld\" (UID: \"3da4e286-2604-498d-91f1-8c88d350eb4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-77dld"
Nov 03 01:31:33.402482 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:33.402450    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fngqw\" (UniqueName: \"kubernetes.io/projected/3da4e286-2604-498d-91f1-8c88d350eb4a-kube-api-access-fngqw\") pod \"maas-api-revoke-sub-e2fc9c5b-77dld\" (UID: \"3da4e286-2604-498d-91f1-8c88d350eb4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-77dld"
Nov 03 01:31:33.402612 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:33.402521    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da4e286-2604-498d-91f1-8c88d350eb4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-77dld\" (UID: \"3da4e286-2604-498d-91f1-8c88d350eb4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-77dld"
Nov 03 01:31:33.402612 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:33.402547    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da4e286-2604-498d-91f1-8c88d350eb4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-77dld\" (UID: \"3da4e286-2604-498d-91f1-8c88d350eb4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-77dld"
Nov 03 01:31:33.403189 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:33.403159    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da4e286-2604-498d-91f1-8c88d350eb4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-77dld\" (UID: \"3da4e286-2604-498d-91f1-8c88d350eb4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-77dld"
Nov 03 01:31:33.404986 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:33.404959    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da4e286-2604-498d-91f1-8c88d350eb4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-77dld\" (UID: \"3da4e286-2604-498d-91f1-8c88d350eb4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-77dld"
Nov 03 01:31:33.429610 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:33.429585    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fngqw\" (UniqueName: \"kubernetes.io/projected/3da4e286-2604-498d-91f1-8c88d350eb4a-kube-api-access-fngqw\") pod \"maas-api-revoke-sub-e2fc9c5b-77dld\" (UID: \"3da4e286-2604-498d-91f1-8c88d350eb4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-77dld"
Nov 03 01:31:33.587645 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:33.587585    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-77dld"
Nov 03 01:31:33.884858 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:33.884833    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-77dld"]
Nov 03 01:31:33.886328 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:31:33.886302    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da4e286_2604_498d_91f1_8c88d350eb4a.slice/crio-aa04c294130383eca3e2de65c11b94bb4563687bfb86cd9f946d6bd1dfe79d0c WatchSource:0}: Error finding container aa04c294130383eca3e2de65c11b94bb4563687bfb86cd9f946d6bd1dfe79d0c: Status 404 returned error can't find the container with id aa04c294130383eca3e2de65c11b94bb4563687bfb86cd9f946d6bd1dfe79d0c
Nov 03 01:31:33.997393 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:33.997361    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-77dld" event={"ID":"3da4e286-2604-498d-91f1-8c88d350eb4a","Type":"ContainerStarted","Data":"aa04c294130383eca3e2de65c11b94bb4563687bfb86cd9f946d6bd1dfe79d0c"}
Nov 03 01:31:35.005622 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:35.005544    5012 generic.go:356] "Generic (PLEG): container finished" podID="3da4e286-2604-498d-91f1-8c88d350eb4a" containerID="886ad9eea6a9f8d3bf31f5fd86125899f9a696abe47121d2fe57f70b19febe68" exitCode=0
Nov 03 01:31:35.005622 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:35.005597    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-77dld" event={"ID":"3da4e286-2604-498d-91f1-8c88d350eb4a","Type":"ContainerDied","Data":"886ad9eea6a9f8d3bf31f5fd86125899f9a696abe47121d2fe57f70b19febe68"}
Nov 03 01:31:36.150700 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:36.150667    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-77dld"
Nov 03 01:31:36.227743 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:36.227714    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3da4e286-2604-498d-91f1-8c88d350eb4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da4e286-2604-498d-91f1-8c88d350eb4a-maas-api-xks-ca\") pod \"3da4e286-2604-498d-91f1-8c88d350eb4a\" (UID: \"3da4e286-2604-498d-91f1-8c88d350eb4a\") "
Nov 03 01:31:36.227936 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:36.227808    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3da4e286-2604-498d-91f1-8c88d350eb4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da4e286-2604-498d-91f1-8c88d350eb4a-maas-api-openshift-ca\") pod \"3da4e286-2604-498d-91f1-8c88d350eb4a\" (UID: \"3da4e286-2604-498d-91f1-8c88d350eb4a\") "
Nov 03 01:31:36.227936 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:36.227878    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3da4e286-2604-498d-91f1-8c88d350eb4a-kube-api-access-fngqw\" (UniqueName: \"kubernetes.io/projected/3da4e286-2604-498d-91f1-8c88d350eb4a-kube-api-access-fngqw\") pod \"3da4e286-2604-498d-91f1-8c88d350eb4a\" (UID: \"3da4e286-2604-498d-91f1-8c88d350eb4a\") "
Nov 03 01:31:36.228196 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:36.228173    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3da4e286-2604-498d-91f1-8c88d350eb4a-maas-api-openshift-ca" pod "3da4e286-2604-498d-91f1-8c88d350eb4a" (UID: "3da4e286-2604-498d-91f1-8c88d350eb4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:31:36.229939 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:36.229899    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3da4e286-2604-498d-91f1-8c88d350eb4a-maas-api-xks-ca" pod "3da4e286-2604-498d-91f1-8c88d350eb4a" (UID: "3da4e286-2604-498d-91f1-8c88d350eb4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:31:36.230012 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:36.229943    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da4e286-2604-498d-91f1-8c88d350eb4a-kube-api-access-fngqw" pod "3da4e286-2604-498d-91f1-8c88d350eb4a" (UID: "3da4e286-2604-498d-91f1-8c88d350eb4a"). InnerVolumeSpecName "kube-api-access-fngqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:31:36.329568 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:36.329493    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da4e286-2604-498d-91f1-8c88d350eb4a-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:31:36.329568 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:36.329527    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fngqw\" (UniqueName: \"kubernetes.io/projected/3da4e286-2604-498d-91f1-8c88d350eb4a-kube-api-access-fngqw\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:31:36.329568 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:36.329543    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da4e286-2604-498d-91f1-8c88d350eb4a-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:31:37.020219 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:37.020189    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-77dld"
Nov 03 01:31:37.020219 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:37.020208    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-77dld" event={"ID":"3da4e286-2604-498d-91f1-8c88d350eb4a","Type":"ContainerDied","Data":"aa04c294130383eca3e2de65c11b94bb4563687bfb86cd9f946d6bd1dfe79d0c"}
Nov 03 01:31:37.020473 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:37.020251    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa04c294130383eca3e2de65c11b94bb4563687bfb86cd9f946d6bd1dfe79d0c"
Nov 03 01:31:37.362074 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:31:37.362042    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:31:43.639426 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:43.639388    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zn96c"]
Nov 03 01:31:43.639869 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:43.639843    5012 cpu_manager.go:414] "containerMap: removing container" podUID="3da4e286-2604-498d-91f1-8c88d350eb4a" containerName="revoke-keys"
Nov 03 01:31:43.639869 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:43.639858    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3da4e286-2604-498d-91f1-8c88d350eb4a" containerName="revoke-keys"
Nov 03 01:31:43.639989 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:43.639963    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="3da4e286-2604-498d-91f1-8c88d350eb4a" containerName="revoke-keys"
Nov 03 01:31:43.643539 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:43.643522    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zn96c"
Nov 03 01:31:43.660441 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:43.660418    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zn96c"]
Nov 03 01:31:43.668945 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:43.666904    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:31:43.668945 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:43.666930    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:31:43.668945 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:43.667009    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:31:43.668945 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:43.667016    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:31:43.690134 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:43.690108    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbpxm\" (UniqueName: \"kubernetes.io/projected/762ca9fc-fe6e-456e-abca-2413bd52896c-kube-api-access-bbpxm\") pod \"maas-api-revoke-sub-5a29efb3-zn96c\" (UID: \"762ca9fc-fe6e-456e-abca-2413bd52896c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zn96c"
Nov 03 01:31:43.690548 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:43.690518    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/762ca9fc-fe6e-456e-abca-2413bd52896c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-zn96c\" (UID: \"762ca9fc-fe6e-456e-abca-2413bd52896c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zn96c"
Nov 03 01:31:43.690656 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:43.690601    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/762ca9fc-fe6e-456e-abca-2413bd52896c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-zn96c\" (UID: \"762ca9fc-fe6e-456e-abca-2413bd52896c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zn96c"
Nov 03 01:31:43.791437 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:43.791400    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbpxm\" (UniqueName: \"kubernetes.io/projected/762ca9fc-fe6e-456e-abca-2413bd52896c-kube-api-access-bbpxm\") pod \"maas-api-revoke-sub-5a29efb3-zn96c\" (UID: \"762ca9fc-fe6e-456e-abca-2413bd52896c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zn96c"
Nov 03 01:31:43.791571 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:43.791493    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/762ca9fc-fe6e-456e-abca-2413bd52896c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-zn96c\" (UID: \"762ca9fc-fe6e-456e-abca-2413bd52896c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zn96c"
Nov 03 01:31:43.791571 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:43.791549    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/762ca9fc-fe6e-456e-abca-2413bd52896c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-zn96c\" (UID: \"762ca9fc-fe6e-456e-abca-2413bd52896c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zn96c"
Nov 03 01:31:43.792265 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:43.792244    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/762ca9fc-fe6e-456e-abca-2413bd52896c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-zn96c\" (UID: \"762ca9fc-fe6e-456e-abca-2413bd52896c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zn96c"
Nov 03 01:31:43.793769 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:43.793752    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/762ca9fc-fe6e-456e-abca-2413bd52896c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-zn96c\" (UID: \"762ca9fc-fe6e-456e-abca-2413bd52896c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zn96c"
Nov 03 01:31:43.808187 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:43.808159    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbpxm\" (UniqueName: \"kubernetes.io/projected/762ca9fc-fe6e-456e-abca-2413bd52896c-kube-api-access-bbpxm\") pod \"maas-api-revoke-sub-5a29efb3-zn96c\" (UID: \"762ca9fc-fe6e-456e-abca-2413bd52896c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zn96c"
Nov 03 01:31:43.961947 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:43.961855    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zn96c"
Nov 03 01:31:44.258038 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:44.258009    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zn96c"]
Nov 03 01:31:44.258968 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:31:44.258894    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762ca9fc_fe6e_456e_abca_2413bd52896c.slice/crio-449b6f3dcdacc72b6555f4933035e8f053cf1efd0c27c2e073cf8f65eb45aeb9 WatchSource:0}: Error finding container 449b6f3dcdacc72b6555f4933035e8f053cf1efd0c27c2e073cf8f65eb45aeb9: Status 404 returned error can't find the container with id 449b6f3dcdacc72b6555f4933035e8f053cf1efd0c27c2e073cf8f65eb45aeb9
Nov 03 01:31:45.074772 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:45.074712    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zn96c" event={"ID":"762ca9fc-fe6e-456e-abca-2413bd52896c","Type":"ContainerStarted","Data":"51d14475d509f5491ac5efaf5f3291001ed81e81eea8c4345b13bd9734f213bf"}
Nov 03 01:31:45.075432 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:45.075401    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zn96c" event={"ID":"762ca9fc-fe6e-456e-abca-2413bd52896c","Type":"ContainerStarted","Data":"449b6f3dcdacc72b6555f4933035e8f053cf1efd0c27c2e073cf8f65eb45aeb9"}
Nov 03 01:31:45.112334 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:45.112275    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zn96c" podStartSLOduration=2.112257977 podStartE2EDuration="2.112257977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:31:45.108171319 +0000 UTC m=+5187.987755363" watchObservedRunningTime="2026-11-03 01:31:45.112257977 +0000 UTC m=+5187.991842021"
Nov 03 01:31:46.108933 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:46.085897    5012 generic.go:356] "Generic (PLEG): container finished" podID="762ca9fc-fe6e-456e-abca-2413bd52896c" containerID="51d14475d509f5491ac5efaf5f3291001ed81e81eea8c4345b13bd9734f213bf" exitCode=0
Nov 03 01:31:46.108933 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:46.086083    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zn96c" event={"ID":"762ca9fc-fe6e-456e-abca-2413bd52896c","Type":"ContainerDied","Data":"51d14475d509f5491ac5efaf5f3291001ed81e81eea8c4345b13bd9734f213bf"}
Nov 03 01:31:47.496033 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:47.495381    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zn96c"
Nov 03 01:31:47.577772 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:47.577742    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/762ca9fc-fe6e-456e-abca-2413bd52896c-kube-api-access-bbpxm\" (UniqueName: \"kubernetes.io/projected/762ca9fc-fe6e-456e-abca-2413bd52896c-kube-api-access-bbpxm\") pod \"762ca9fc-fe6e-456e-abca-2413bd52896c\" (UID: \"762ca9fc-fe6e-456e-abca-2413bd52896c\") "
Nov 03 01:31:47.577994 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:47.577826    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/762ca9fc-fe6e-456e-abca-2413bd52896c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/762ca9fc-fe6e-456e-abca-2413bd52896c-maas-api-xks-ca\") pod \"762ca9fc-fe6e-456e-abca-2413bd52896c\" (UID: \"762ca9fc-fe6e-456e-abca-2413bd52896c\") "
Nov 03 01:31:47.577994 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:47.577868    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/762ca9fc-fe6e-456e-abca-2413bd52896c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/762ca9fc-fe6e-456e-abca-2413bd52896c-maas-api-openshift-ca\") pod \"762ca9fc-fe6e-456e-abca-2413bd52896c\" (UID: \"762ca9fc-fe6e-456e-abca-2413bd52896c\") "
Nov 03 01:31:47.578371 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:47.578342    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/762ca9fc-fe6e-456e-abca-2413bd52896c-maas-api-openshift-ca" pod "762ca9fc-fe6e-456e-abca-2413bd52896c" (UID: "762ca9fc-fe6e-456e-abca-2413bd52896c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:31:47.580024 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:47.579991    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/762ca9fc-fe6e-456e-abca-2413bd52896c-maas-api-xks-ca" pod "762ca9fc-fe6e-456e-abca-2413bd52896c" (UID: "762ca9fc-fe6e-456e-abca-2413bd52896c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:31:47.580122 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:47.580011    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762ca9fc-fe6e-456e-abca-2413bd52896c-kube-api-access-bbpxm" pod "762ca9fc-fe6e-456e-abca-2413bd52896c" (UID: "762ca9fc-fe6e-456e-abca-2413bd52896c"). InnerVolumeSpecName "kube-api-access-bbpxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:31:47.679131 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:47.679098    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbpxm\" (UniqueName: \"kubernetes.io/projected/762ca9fc-fe6e-456e-abca-2413bd52896c-kube-api-access-bbpxm\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:31:47.679131 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:47.679129    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/762ca9fc-fe6e-456e-abca-2413bd52896c-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:31:47.679275 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:47.679143    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/762ca9fc-fe6e-456e-abca-2413bd52896c-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:31:48.121707 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:48.121666    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zn96c" event={"ID":"762ca9fc-fe6e-456e-abca-2413bd52896c","Type":"ContainerDied","Data":"449b6f3dcdacc72b6555f4933035e8f053cf1efd0c27c2e073cf8f65eb45aeb9"}
Nov 03 01:31:48.121707 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:48.121694    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zn96c"
Nov 03 01:31:48.121949 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:48.121727    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="449b6f3dcdacc72b6555f4933035e8f053cf1efd0c27c2e073cf8f65eb45aeb9"
Nov 03 01:31:59.829979 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:59.829944    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nptmb"]
Nov 03 01:31:59.841473 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:59.841443    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nptmb"]
Nov 03 01:31:59.877727 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:59.877690    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c5z"]
Nov 03 01:31:59.878132 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:59.878118    5012 cpu_manager.go:414] "containerMap: removing container" podUID="762ca9fc-fe6e-456e-abca-2413bd52896c" containerName="revoke-keys"
Nov 03 01:31:59.878182 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:59.878133    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="762ca9fc-fe6e-456e-abca-2413bd52896c" containerName="revoke-keys"
Nov 03 01:31:59.878234 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:59.878221    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="762ca9fc-fe6e-456e-abca-2413bd52896c" containerName="revoke-keys"
Nov 03 01:31:59.881548 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:59.881530    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c5z"
Nov 03 01:31:59.890575 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:59.890548    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6301cfd-fecd-460a-8f62-26aedce4edd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b2c5z\" (UID: \"d6301cfd-fecd-460a-8f62-26aedce4edd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c5z"
Nov 03 01:31:59.890701 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:59.890619    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6301cfd-fecd-460a-8f62-26aedce4edd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b2c5z\" (UID: \"d6301cfd-fecd-460a-8f62-26aedce4edd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c5z"
Nov 03 01:31:59.890701 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:59.890663    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7drg\" (UniqueName: \"kubernetes.io/projected/d6301cfd-fecd-460a-8f62-26aedce4edd5-kube-api-access-z7drg\") pod \"maas-api-revoke-sub-3d4156f1-b2c5z\" (UID: \"d6301cfd-fecd-460a-8f62-26aedce4edd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c5z"
Nov 03 01:31:59.897767 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:59.897741    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:31:59.897922 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:59.897777    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:31:59.898044 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:59.897800    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:31:59.898044 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:59.897962    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:31:59.901653 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:59.901630    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c5z"]
Nov 03 01:31:59.992105 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:59.992073    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6301cfd-fecd-460a-8f62-26aedce4edd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b2c5z\" (UID: \"d6301cfd-fecd-460a-8f62-26aedce4edd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c5z"
Nov 03 01:31:59.992282 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:59.992121    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7drg\" (UniqueName: \"kubernetes.io/projected/d6301cfd-fecd-460a-8f62-26aedce4edd5-kube-api-access-z7drg\") pod \"maas-api-revoke-sub-3d4156f1-b2c5z\" (UID: \"d6301cfd-fecd-460a-8f62-26aedce4edd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c5z"
Nov 03 01:31:59.992282 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:59.992188    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6301cfd-fecd-460a-8f62-26aedce4edd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b2c5z\" (UID: \"d6301cfd-fecd-460a-8f62-26aedce4edd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c5z"
Nov 03 01:31:59.992905 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:59.992878    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6301cfd-fecd-460a-8f62-26aedce4edd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b2c5z\" (UID: \"d6301cfd-fecd-460a-8f62-26aedce4edd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c5z"
Nov 03 01:31:59.994462 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:31:59.994429    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6301cfd-fecd-460a-8f62-26aedce4edd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b2c5z\" (UID: \"d6301cfd-fecd-460a-8f62-26aedce4edd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c5z"
Nov 03 01:32:00.008928 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:00.008881    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7drg\" (UniqueName: \"kubernetes.io/projected/d6301cfd-fecd-460a-8f62-26aedce4edd5-kube-api-access-z7drg\") pod \"maas-api-revoke-sub-3d4156f1-b2c5z\" (UID: \"d6301cfd-fecd-460a-8f62-26aedce4edd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c5z"
Nov 03 01:32:00.195169 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:00.195132    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c5z"
Nov 03 01:32:00.487521 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:00.487491    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c5z"]
Nov 03 01:32:00.488987 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:32:00.488961    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6301cfd_fecd_460a_8f62_26aedce4edd5.slice/crio-85af44a2033042d36ce5f26505e73b0ab396afb8734627cd9726f6a3251d8742 WatchSource:0}: Error finding container 85af44a2033042d36ce5f26505e73b0ab396afb8734627cd9726f6a3251d8742: Status 404 returned error can't find the container with id 85af44a2033042d36ce5f26505e73b0ab396afb8734627cd9726f6a3251d8742
Nov 03 01:32:01.204979 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:01.204940    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c5z" event={"ID":"d6301cfd-fecd-460a-8f62-26aedce4edd5","Type":"ContainerStarted","Data":"d7736154527bbbdd1704ae210d4d74690c6daf9f76ad40678d198190a2289bd2"}
Nov 03 01:32:01.205354 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:01.205003    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c5z" event={"ID":"d6301cfd-fecd-460a-8f62-26aedce4edd5","Type":"ContainerStarted","Data":"85af44a2033042d36ce5f26505e73b0ab396afb8734627cd9726f6a3251d8742"}
Nov 03 01:32:01.239344 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:01.239276    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c5z" podStartSLOduration=2.239259932 podStartE2EDuration="2.239259932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:32:01.238364414 +0000 UTC m=+5204.117948456" watchObservedRunningTime="2026-11-03 01:32:01.239259932 +0000 UTC m=+5204.118843974"
Nov 03 01:32:01.242939 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:01.242901    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e27b88ce-be07-4e3e-b127-c5bc29f1cb9e" path="/var/lib/kubelet/pods/e27b88ce-be07-4e3e-b127-c5bc29f1cb9e/volumes"
Nov 03 01:32:02.212832 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:02.212798    5012 generic.go:356] "Generic (PLEG): container finished" podID="d6301cfd-fecd-460a-8f62-26aedce4edd5" containerID="d7736154527bbbdd1704ae210d4d74690c6daf9f76ad40678d198190a2289bd2" exitCode=0
Nov 03 01:32:02.213238 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:02.212882    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c5z" event={"ID":"d6301cfd-fecd-460a-8f62-26aedce4edd5","Type":"ContainerDied","Data":"d7736154527bbbdd1704ae210d4d74690c6daf9f76ad40678d198190a2289bd2"}
Nov 03 01:32:03.352747 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:03.352724    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c5z"
Nov 03 01:32:03.421653 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:03.421619    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6301cfd-fecd-460a-8f62-26aedce4edd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6301cfd-fecd-460a-8f62-26aedce4edd5-maas-api-xks-ca\") pod \"d6301cfd-fecd-460a-8f62-26aedce4edd5\" (UID: \"d6301cfd-fecd-460a-8f62-26aedce4edd5\") "
Nov 03 01:32:03.421888 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:03.421685    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6301cfd-fecd-460a-8f62-26aedce4edd5-kube-api-access-z7drg\" (UniqueName: \"kubernetes.io/projected/d6301cfd-fecd-460a-8f62-26aedce4edd5-kube-api-access-z7drg\") pod \"d6301cfd-fecd-460a-8f62-26aedce4edd5\" (UID: \"d6301cfd-fecd-460a-8f62-26aedce4edd5\") "
Nov 03 01:32:03.421888 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:03.421743    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6301cfd-fecd-460a-8f62-26aedce4edd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6301cfd-fecd-460a-8f62-26aedce4edd5-maas-api-openshift-ca\") pod \"d6301cfd-fecd-460a-8f62-26aedce4edd5\" (UID: \"d6301cfd-fecd-460a-8f62-26aedce4edd5\") "
Nov 03 01:32:03.422147 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:03.422122    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6301cfd-fecd-460a-8f62-26aedce4edd5-maas-api-openshift-ca" pod "d6301cfd-fecd-460a-8f62-26aedce4edd5" (UID: "d6301cfd-fecd-460a-8f62-26aedce4edd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:32:03.423776 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:03.423753    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6301cfd-fecd-460a-8f62-26aedce4edd5-maas-api-xks-ca" pod "d6301cfd-fecd-460a-8f62-26aedce4edd5" (UID: "d6301cfd-fecd-460a-8f62-26aedce4edd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:32:03.423876 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:03.423807    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6301cfd-fecd-460a-8f62-26aedce4edd5-kube-api-access-z7drg" pod "d6301cfd-fecd-460a-8f62-26aedce4edd5" (UID: "d6301cfd-fecd-460a-8f62-26aedce4edd5"). InnerVolumeSpecName "kube-api-access-z7drg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:32:03.522970 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:03.522879    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6301cfd-fecd-460a-8f62-26aedce4edd5-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:32:03.522970 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:03.522929    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7drg\" (UniqueName: \"kubernetes.io/projected/d6301cfd-fecd-460a-8f62-26aedce4edd5-kube-api-access-z7drg\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:32:03.522970 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:03.522940    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6301cfd-fecd-460a-8f62-26aedce4edd5-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:32:04.227144 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:04.227109    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c5z"
Nov 03 01:32:04.227346 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:04.227138    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c5z" event={"ID":"d6301cfd-fecd-460a-8f62-26aedce4edd5","Type":"ContainerDied","Data":"85af44a2033042d36ce5f26505e73b0ab396afb8734627cd9726f6a3251d8742"}
Nov 03 01:32:04.227346 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:04.227188    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85af44a2033042d36ce5f26505e73b0ab396afb8734627cd9726f6a3251d8742"
Nov 03 01:32:07.362477 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:32:07.362443    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:32:22.220439 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:22.220402    5012 scope.go:122] "RemoveContainer" containerID="a31ffea9d36469b3e3ac2d90ec9c653862513695bc49153ce70bc88f598151a0"
Nov 03 01:32:37.365334 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:32:37.365219    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:32:55.024609 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:55.024568    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-cqlms"]
Nov 03 01:32:55.034577 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:55.034550    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-cqlms"]
Nov 03 01:32:55.243410 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:32:55.243373    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2194817d-464f-47aa-aaad-2b751cd0d2d1" path="/var/lib/kubelet/pods/2194817d-464f-47aa-aaad-2b751cd0d2d1/volumes"
Nov 03 01:33:05.808003 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:05.807753    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75559f53-csjhd"]
Nov 03 01:33:05.809082 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:05.809056    5012 cpu_manager.go:414] "containerMap: removing container" podUID="d6301cfd-fecd-460a-8f62-26aedce4edd5" containerName="revoke-keys"
Nov 03 01:33:05.809280 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:05.809262    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6301cfd-fecd-460a-8f62-26aedce4edd5" containerName="revoke-keys"
Nov 03 01:33:05.809609 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:05.809591    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6301cfd-fecd-460a-8f62-26aedce4edd5" containerName="revoke-keys"
Nov 03 01:33:05.813988 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:05.813963    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75559f53-csjhd"
Nov 03 01:33:05.826490 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:05.826452    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75559f53-csjhd"]
Nov 03 01:33:05.833739 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:05.833376    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:33:05.833739 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:05.833592    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:33:05.833739 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:05.833687    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:33:05.837175 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:05.837153    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:33:05.913055 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:05.913025    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a98939a9-969d-41c3-99c5-6526d48f2f08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75559f53-csjhd\" (UID: \"a98939a9-969d-41c3-99c5-6526d48f2f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75559f53-csjhd"
Nov 03 01:33:05.913208 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:05.913098    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a98939a9-969d-41c3-99c5-6526d48f2f08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75559f53-csjhd\" (UID: \"a98939a9-969d-41c3-99c5-6526d48f2f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75559f53-csjhd"
Nov 03 01:33:05.913208 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:05.913183    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxnln\" (UniqueName: \"kubernetes.io/projected/a98939a9-969d-41c3-99c5-6526d48f2f08-kube-api-access-bxnln\") pod \"maas-api-revoke-sub-75559f53-csjhd\" (UID: \"a98939a9-969d-41c3-99c5-6526d48f2f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75559f53-csjhd"
Nov 03 01:33:06.014563 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:06.014533    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a98939a9-969d-41c3-99c5-6526d48f2f08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75559f53-csjhd\" (UID: \"a98939a9-969d-41c3-99c5-6526d48f2f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75559f53-csjhd"
Nov 03 01:33:06.014743 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:06.014617    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a98939a9-969d-41c3-99c5-6526d48f2f08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75559f53-csjhd\" (UID: \"a98939a9-969d-41c3-99c5-6526d48f2f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75559f53-csjhd"
Nov 03 01:33:06.014743 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:06.014670    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxnln\" (UniqueName: \"kubernetes.io/projected/a98939a9-969d-41c3-99c5-6526d48f2f08-kube-api-access-bxnln\") pod \"maas-api-revoke-sub-75559f53-csjhd\" (UID: \"a98939a9-969d-41c3-99c5-6526d48f2f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75559f53-csjhd"
Nov 03 01:33:06.015404 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:06.015379    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a98939a9-969d-41c3-99c5-6526d48f2f08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75559f53-csjhd\" (UID: \"a98939a9-969d-41c3-99c5-6526d48f2f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75559f53-csjhd"
Nov 03 01:33:06.017032 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:06.017007    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a98939a9-969d-41c3-99c5-6526d48f2f08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75559f53-csjhd\" (UID: \"a98939a9-969d-41c3-99c5-6526d48f2f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75559f53-csjhd"
Nov 03 01:33:06.032067 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:06.032036    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxnln\" (UniqueName: \"kubernetes.io/projected/a98939a9-969d-41c3-99c5-6526d48f2f08-kube-api-access-bxnln\") pod \"maas-api-revoke-sub-75559f53-csjhd\" (UID: \"a98939a9-969d-41c3-99c5-6526d48f2f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75559f53-csjhd"
Nov 03 01:33:06.130001 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:06.129969    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75559f53-csjhd"
Nov 03 01:33:06.422176 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:06.422147    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75559f53-csjhd"]
Nov 03 01:33:06.423413 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:33:06.423384    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98939a9_969d_41c3_99c5_6526d48f2f08.slice/crio-ee655e311a239dae307f73377f85e9ab19e14ad3d5a13db63f5e2ffd07650061 WatchSource:0}: Error finding container ee655e311a239dae307f73377f85e9ab19e14ad3d5a13db63f5e2ffd07650061: Status 404 returned error can't find the container with id ee655e311a239dae307f73377f85e9ab19e14ad3d5a13db63f5e2ffd07650061
Nov 03 01:33:06.654228 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:06.654194    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75559f53-csjhd" event={"ID":"a98939a9-969d-41c3-99c5-6526d48f2f08","Type":"ContainerStarted","Data":"ee655e311a239dae307f73377f85e9ab19e14ad3d5a13db63f5e2ffd07650061"}
Nov 03 01:33:07.363032 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:33:07.362998    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:33:07.662108 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:07.662071    5012 generic.go:356] "Generic (PLEG): container finished" podID="a98939a9-969d-41c3-99c5-6526d48f2f08" containerID="e68984f592263199fcdd01234941949937e75b9fcd98e3fa5b0b15e94747ecd3" exitCode=0
Nov 03 01:33:07.662317 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:07.662155    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75559f53-csjhd" event={"ID":"a98939a9-969d-41c3-99c5-6526d48f2f08","Type":"ContainerDied","Data":"e68984f592263199fcdd01234941949937e75b9fcd98e3fa5b0b15e94747ecd3"}
Nov 03 01:33:08.804641 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:08.804615    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75559f53-csjhd"
Nov 03 01:33:08.940375 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:08.940292    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a98939a9-969d-41c3-99c5-6526d48f2f08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a98939a9-969d-41c3-99c5-6526d48f2f08-maas-api-openshift-ca\") pod \"a98939a9-969d-41c3-99c5-6526d48f2f08\" (UID: \"a98939a9-969d-41c3-99c5-6526d48f2f08\") "
Nov 03 01:33:08.940375 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:08.940363    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a98939a9-969d-41c3-99c5-6526d48f2f08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a98939a9-969d-41c3-99c5-6526d48f2f08-maas-api-xks-ca\") pod \"a98939a9-969d-41c3-99c5-6526d48f2f08\" (UID: \"a98939a9-969d-41c3-99c5-6526d48f2f08\") "
Nov 03 01:33:08.940582 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:08.940393    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a98939a9-969d-41c3-99c5-6526d48f2f08-kube-api-access-bxnln\" (UniqueName: \"kubernetes.io/projected/a98939a9-969d-41c3-99c5-6526d48f2f08-kube-api-access-bxnln\") pod \"a98939a9-969d-41c3-99c5-6526d48f2f08\" (UID: \"a98939a9-969d-41c3-99c5-6526d48f2f08\") "
Nov 03 01:33:08.940653 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:08.940630    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a98939a9-969d-41c3-99c5-6526d48f2f08-maas-api-openshift-ca" pod "a98939a9-969d-41c3-99c5-6526d48f2f08" (UID: "a98939a9-969d-41c3-99c5-6526d48f2f08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:33:08.940797 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:08.940783    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a98939a9-969d-41c3-99c5-6526d48f2f08-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:33:08.942517 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:08.942490    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a98939a9-969d-41c3-99c5-6526d48f2f08-kube-api-access-bxnln" pod "a98939a9-969d-41c3-99c5-6526d48f2f08" (UID: "a98939a9-969d-41c3-99c5-6526d48f2f08"). InnerVolumeSpecName "kube-api-access-bxnln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:33:08.942517 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:08.942503    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a98939a9-969d-41c3-99c5-6526d48f2f08-maas-api-xks-ca" pod "a98939a9-969d-41c3-99c5-6526d48f2f08" (UID: "a98939a9-969d-41c3-99c5-6526d48f2f08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:33:09.042353 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:09.042321    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a98939a9-969d-41c3-99c5-6526d48f2f08-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:33:09.042353 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:09.042352    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxnln\" (UniqueName: \"kubernetes.io/projected/a98939a9-969d-41c3-99c5-6526d48f2f08-kube-api-access-bxnln\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:33:09.676465 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:09.676433    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75559f53-csjhd"
Nov 03 01:33:09.676465 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:09.676445    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75559f53-csjhd" event={"ID":"a98939a9-969d-41c3-99c5-6526d48f2f08","Type":"ContainerDied","Data":"ee655e311a239dae307f73377f85e9ab19e14ad3d5a13db63f5e2ffd07650061"}
Nov 03 01:33:09.676688 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:09.676488    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee655e311a239dae307f73377f85e9ab19e14ad3d5a13db63f5e2ffd07650061"
Nov 03 01:33:18.027492 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:18.027457    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlvjt"]
Nov 03 01:33:18.037419 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:18.037387    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlvjt"]
Nov 03 01:33:19.243009 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:19.242903    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="869b15c7-0285-4c63-b2d3-3cd538662b56" path="/var/lib/kubelet/pods/869b15c7-0285-4c63-b2d3-3cd538662b56/volumes"
Nov 03 01:33:37.362180 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:33:37.362131    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:33:38.032207 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:38.032170    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fkgtf"]
Nov 03 01:33:38.042005 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:38.041972    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fkgtf"]
Nov 03 01:33:39.242711 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:39.242676    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f193d66-6bbe-4c27-b9f0-0df1f0605af1" path="/var/lib/kubelet/pods/6f193d66-6bbe-4c27-b9f0-0df1f0605af1/volumes"
Nov 03 01:33:49.025799 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:49.025761    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x98rl"]
Nov 03 01:33:49.035030 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:49.035004    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x98rl"]
Nov 03 01:33:49.242753 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:33:49.242716    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82206537-1719-4b8c-896e-6d84a680f7af" path="/var/lib/kubelet/pods/82206537-1719-4b8c-896e-6d84a680f7af/volumes"
Nov 03 01:34:06.567181 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:06.567127    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p"]
Nov 03 01:34:06.568553 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:06.568524    5012 cpu_manager.go:414] "containerMap: removing container" podUID="a98939a9-969d-41c3-99c5-6526d48f2f08" containerName="revoke-keys"
Nov 03 01:34:06.568722 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:06.568707    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a98939a9-969d-41c3-99c5-6526d48f2f08" containerName="revoke-keys"
Nov 03 01:34:06.568999 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:06.568982    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="a98939a9-969d-41c3-99c5-6526d48f2f08" containerName="revoke-keys"
Nov 03 01:34:06.587307 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:06.573272    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p"
Nov 03 01:34:06.595085 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:06.594898    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:34:06.595372 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:06.595352    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:34:06.596681 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:06.596659    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p"]
Nov 03 01:34:06.653776 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:06.653735    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:34:06.653776 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:06.653756    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:34:06.695816 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:06.695776    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70365689-86e3-4e35-9849-3648a955550d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p\" (UID: \"70365689-86e3-4e35-9849-3648a955550d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p"
Nov 03 01:34:06.696039 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:06.695826    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5tp4\" (UniqueName: \"kubernetes.io/projected/70365689-86e3-4e35-9849-3648a955550d-kube-api-access-h5tp4\") pod \"maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p\" (UID: \"70365689-86e3-4e35-9849-3648a955550d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p"
Nov 03 01:34:06.696039 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:06.695965    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70365689-86e3-4e35-9849-3648a955550d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p\" (UID: \"70365689-86e3-4e35-9849-3648a955550d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p"
Nov 03 01:34:06.797082 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:06.797045    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70365689-86e3-4e35-9849-3648a955550d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p\" (UID: \"70365689-86e3-4e35-9849-3648a955550d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p"
Nov 03 01:34:06.797267 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:06.797170    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70365689-86e3-4e35-9849-3648a955550d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p\" (UID: \"70365689-86e3-4e35-9849-3648a955550d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p"
Nov 03 01:34:06.797267 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:06.797214    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5tp4\" (UniqueName: \"kubernetes.io/projected/70365689-86e3-4e35-9849-3648a955550d-kube-api-access-h5tp4\") pod \"maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p\" (UID: \"70365689-86e3-4e35-9849-3648a955550d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p"
Nov 03 01:34:06.797849 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:06.797825    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70365689-86e3-4e35-9849-3648a955550d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p\" (UID: \"70365689-86e3-4e35-9849-3648a955550d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p"
Nov 03 01:34:06.799464 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:06.799441    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70365689-86e3-4e35-9849-3648a955550d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p\" (UID: \"70365689-86e3-4e35-9849-3648a955550d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p"
Nov 03 01:34:06.815244 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:06.815220    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5tp4\" (UniqueName: \"kubernetes.io/projected/70365689-86e3-4e35-9849-3648a955550d-kube-api-access-h5tp4\") pod \"maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p\" (UID: \"70365689-86e3-4e35-9849-3648a955550d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p"
Nov 03 01:34:06.905349 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:06.905311    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p"
Nov 03 01:34:07.211827 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:07.211797    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p"]
Nov 03 01:34:07.212978 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:34:07.212947    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70365689_86e3_4e35_9849_3648a955550d.slice/crio-c72293e95087bff53a18645f9828043cce1e1769d5af15f52fbfe072a8eaaf29 WatchSource:0}: Error finding container c72293e95087bff53a18645f9828043cce1e1769d5af15f52fbfe072a8eaaf29: Status 404 returned error can't find the container with id c72293e95087bff53a18645f9828043cce1e1769d5af15f52fbfe072a8eaaf29
Nov 03 01:34:07.362978 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:34:07.362936    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:34:08.054090 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:08.054050    5012 generic.go:356] "Generic (PLEG): container finished" podID="70365689-86e3-4e35-9849-3648a955550d" containerID="19e2eebc21a73590c19ab05015ebdcc8c038c44b1b814eda85f28657b4bea639" exitCode=0
Nov 03 01:34:08.054497 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:08.054142    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p" event={"ID":"70365689-86e3-4e35-9849-3648a955550d","Type":"ContainerDied","Data":"19e2eebc21a73590c19ab05015ebdcc8c038c44b1b814eda85f28657b4bea639"}
Nov 03 01:34:08.054497 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:08.054224    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p" event={"ID":"70365689-86e3-4e35-9849-3648a955550d","Type":"ContainerStarted","Data":"c72293e95087bff53a18645f9828043cce1e1769d5af15f52fbfe072a8eaaf29"}
Nov 03 01:34:09.199413 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:09.199383    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p"
Nov 03 01:34:09.323829 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:09.323721    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70365689-86e3-4e35-9849-3648a955550d-kube-api-access-h5tp4\" (UniqueName: \"kubernetes.io/projected/70365689-86e3-4e35-9849-3648a955550d-kube-api-access-h5tp4\") pod \"70365689-86e3-4e35-9849-3648a955550d\" (UID: \"70365689-86e3-4e35-9849-3648a955550d\") "
Nov 03 01:34:09.323829 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:09.323791    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70365689-86e3-4e35-9849-3648a955550d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70365689-86e3-4e35-9849-3648a955550d-maas-api-xks-ca\") pod \"70365689-86e3-4e35-9849-3648a955550d\" (UID: \"70365689-86e3-4e35-9849-3648a955550d\") "
Nov 03 01:34:09.324110 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:09.323889    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70365689-86e3-4e35-9849-3648a955550d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70365689-86e3-4e35-9849-3648a955550d-maas-api-openshift-ca\") pod \"70365689-86e3-4e35-9849-3648a955550d\" (UID: \"70365689-86e3-4e35-9849-3648a955550d\") "
Nov 03 01:34:09.324317 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:09.324293    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70365689-86e3-4e35-9849-3648a955550d-maas-api-openshift-ca" pod "70365689-86e3-4e35-9849-3648a955550d" (UID: "70365689-86e3-4e35-9849-3648a955550d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:34:09.326076 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:09.326049    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70365689-86e3-4e35-9849-3648a955550d-maas-api-xks-ca" pod "70365689-86e3-4e35-9849-3648a955550d" (UID: "70365689-86e3-4e35-9849-3648a955550d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:34:09.326222 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:09.326079    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70365689-86e3-4e35-9849-3648a955550d-kube-api-access-h5tp4" pod "70365689-86e3-4e35-9849-3648a955550d" (UID: "70365689-86e3-4e35-9849-3648a955550d"). InnerVolumeSpecName "kube-api-access-h5tp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:34:09.425288 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:09.425247    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5tp4\" (UniqueName: \"kubernetes.io/projected/70365689-86e3-4e35-9849-3648a955550d-kube-api-access-h5tp4\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:34:09.425288 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:09.425286    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70365689-86e3-4e35-9849-3648a955550d-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:34:09.425504 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:09.425299    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70365689-86e3-4e35-9849-3648a955550d-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:34:10.067522 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:10.067488    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p"
Nov 03 01:34:10.067522 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:10.067512    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p" event={"ID":"70365689-86e3-4e35-9849-3648a955550d","Type":"ContainerDied","Data":"c72293e95087bff53a18645f9828043cce1e1769d5af15f52fbfe072a8eaaf29"}
Nov 03 01:34:10.067794 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:10.067558    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c72293e95087bff53a18645f9828043cce1e1769d5af15f52fbfe072a8eaaf29"
Nov 03 01:34:16.026488 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:16.026444    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8sjj9"]
Nov 03 01:34:16.036212 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:16.036183    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8sjj9"]
Nov 03 01:34:17.242309 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:17.242274    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d48fef83-16a3-45bd-ac50-108ba16bb4ec" path="/var/lib/kubelet/pods/d48fef83-16a3-45bd-ac50-108ba16bb4ec/volumes"
Nov 03 01:34:19.290789 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:19.290759    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p"]
Nov 03 01:34:19.301033 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:19.301008    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-462c10-k7w8p"]
Nov 03 01:34:20.983210 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:20.983175    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c5z"]
Nov 03 01:34:20.993714 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:20.993684    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b2c5z"]
Nov 03 01:34:21.031518 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:21.031493    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgw2k"]
Nov 03 01:34:21.031974 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:21.031960    5012 cpu_manager.go:414] "containerMap: removing container" podUID="70365689-86e3-4e35-9849-3648a955550d" containerName="revoke-keys"
Nov 03 01:34:21.032023 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:21.031974    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70365689-86e3-4e35-9849-3648a955550d" containerName="revoke-keys"
Nov 03 01:34:21.032061 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:21.032033    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="70365689-86e3-4e35-9849-3648a955550d" containerName="revoke-keys"
Nov 03 01:34:21.036469 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:21.036449    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgw2k"
Nov 03 01:34:21.050063 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:21.050042    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:34:21.050162 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:21.050107    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:34:21.050379 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:21.050359    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:34:21.050540 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:21.050522    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:34:21.054356 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:21.054335    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgw2k"]
Nov 03 01:34:21.157639 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:21.157607    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wzbs\" (UniqueName: \"kubernetes.io/projected/4f93d2fa-4486-4f9c-973e-7e0db7272176-kube-api-access-5wzbs\") pod \"maas-api-revoke-sub-3d4156f1-kgw2k\" (UID: \"4f93d2fa-4486-4f9c-973e-7e0db7272176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgw2k"
Nov 03 01:34:21.157793 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:21.157683    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f93d2fa-4486-4f9c-973e-7e0db7272176-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kgw2k\" (UID: \"4f93d2fa-4486-4f9c-973e-7e0db7272176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgw2k"
Nov 03 01:34:21.157793 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:21.157744    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f93d2fa-4486-4f9c-973e-7e0db7272176-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kgw2k\" (UID: \"4f93d2fa-4486-4f9c-973e-7e0db7272176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgw2k"
Nov 03 01:34:21.244087 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:21.244020    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70365689-86e3-4e35-9849-3648a955550d" path="/var/lib/kubelet/pods/70365689-86e3-4e35-9849-3648a955550d/volumes"
Nov 03 01:34:21.244449 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:21.244435    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6301cfd-fecd-460a-8f62-26aedce4edd5" path="/var/lib/kubelet/pods/d6301cfd-fecd-460a-8f62-26aedce4edd5/volumes"
Nov 03 01:34:21.258394 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:21.258366    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f93d2fa-4486-4f9c-973e-7e0db7272176-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kgw2k\" (UID: \"4f93d2fa-4486-4f9c-973e-7e0db7272176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgw2k"
Nov 03 01:34:21.258531 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:21.258439    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wzbs\" (UniqueName: \"kubernetes.io/projected/4f93d2fa-4486-4f9c-973e-7e0db7272176-kube-api-access-5wzbs\") pod \"maas-api-revoke-sub-3d4156f1-kgw2k\" (UID: \"4f93d2fa-4486-4f9c-973e-7e0db7272176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgw2k"
Nov 03 01:34:21.258603 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:21.258538    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f93d2fa-4486-4f9c-973e-7e0db7272176-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kgw2k\" (UID: \"4f93d2fa-4486-4f9c-973e-7e0db7272176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgw2k"
Nov 03 01:34:21.259178 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:21.259155    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f93d2fa-4486-4f9c-973e-7e0db7272176-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kgw2k\" (UID: \"4f93d2fa-4486-4f9c-973e-7e0db7272176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgw2k"
Nov 03 01:34:21.260788 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:21.260762    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f93d2fa-4486-4f9c-973e-7e0db7272176-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kgw2k\" (UID: \"4f93d2fa-4486-4f9c-973e-7e0db7272176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgw2k"
Nov 03 01:34:21.275366 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:21.275342    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wzbs\" (UniqueName: \"kubernetes.io/projected/4f93d2fa-4486-4f9c-973e-7e0db7272176-kube-api-access-5wzbs\") pod \"maas-api-revoke-sub-3d4156f1-kgw2k\" (UID: \"4f93d2fa-4486-4f9c-973e-7e0db7272176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgw2k"
Nov 03 01:34:21.349482 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:21.349450    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgw2k"
Nov 03 01:34:21.644282 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:21.644255    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgw2k"]
Nov 03 01:34:21.645589 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:34:21.645566    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f93d2fa_4486_4f9c_973e_7e0db7272176.slice/crio-a73eea18906ee538b18caea2459f484c28299b9eb90c1c5f16ff90f79a7ec53c WatchSource:0}: Error finding container a73eea18906ee538b18caea2459f484c28299b9eb90c1c5f16ff90f79a7ec53c: Status 404 returned error can't find the container with id a73eea18906ee538b18caea2459f484c28299b9eb90c1c5f16ff90f79a7ec53c
Nov 03 01:34:22.165029 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:22.164953    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgw2k" event={"ID":"4f93d2fa-4486-4f9c-973e-7e0db7272176","Type":"ContainerStarted","Data":"a08c5e04f8ae8f813e9b2f92daa20bc28565a352bae4494f8b8cde142e7b6d4b"}
Nov 03 01:34:22.165029 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:22.165012    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgw2k" event={"ID":"4f93d2fa-4486-4f9c-973e-7e0db7272176","Type":"ContainerStarted","Data":"a73eea18906ee538b18caea2459f484c28299b9eb90c1c5f16ff90f79a7ec53c"}
Nov 03 01:34:22.199473 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:22.199421    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgw2k" podStartSLOduration=1.199406954 podStartE2EDuration="1.199406954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:34:22.197523392 +0000 UTC m=+5345.077107435" watchObservedRunningTime="2026-11-03 01:34:22.199406954 +0000 UTC m=+5345.078990996"
Nov 03 01:34:22.258233 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:22.258209    5012 scope.go:122] "RemoveContainer" containerID="2b9b1b2a70b544674166e7b3aa15fff64429c2af7ecfaa8f352362353ae773bd"
Nov 03 01:34:22.273741 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:22.273719    5012 scope.go:122] "RemoveContainer" containerID="31183769b2ecbfcb9429356528f18358bc4c5817259ca6c32d31a987da006466"
Nov 03 01:34:23.173264 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:23.173223    5012 generic.go:356] "Generic (PLEG): container finished" podID="4f93d2fa-4486-4f9c-973e-7e0db7272176" containerID="a08c5e04f8ae8f813e9b2f92daa20bc28565a352bae4494f8b8cde142e7b6d4b" exitCode=0
Nov 03 01:34:23.173653 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:23.173306    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgw2k" event={"ID":"4f93d2fa-4486-4f9c-973e-7e0db7272176","Type":"ContainerDied","Data":"a08c5e04f8ae8f813e9b2f92daa20bc28565a352bae4494f8b8cde142e7b6d4b"}
Nov 03 01:34:24.319061 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:24.319034    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgw2k"
Nov 03 01:34:24.488535 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:24.488467    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f93d2fa-4486-4f9c-973e-7e0db7272176-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f93d2fa-4486-4f9c-973e-7e0db7272176-maas-api-xks-ca\") pod \"4f93d2fa-4486-4f9c-973e-7e0db7272176\" (UID: \"4f93d2fa-4486-4f9c-973e-7e0db7272176\") "
Nov 03 01:34:24.488674 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:24.488564    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f93d2fa-4486-4f9c-973e-7e0db7272176-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f93d2fa-4486-4f9c-973e-7e0db7272176-maas-api-openshift-ca\") pod \"4f93d2fa-4486-4f9c-973e-7e0db7272176\" (UID: \"4f93d2fa-4486-4f9c-973e-7e0db7272176\") "
Nov 03 01:34:24.488674 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:24.488617    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f93d2fa-4486-4f9c-973e-7e0db7272176-kube-api-access-5wzbs\" (UniqueName: \"kubernetes.io/projected/4f93d2fa-4486-4f9c-973e-7e0db7272176-kube-api-access-5wzbs\") pod \"4f93d2fa-4486-4f9c-973e-7e0db7272176\" (UID: \"4f93d2fa-4486-4f9c-973e-7e0db7272176\") "
Nov 03 01:34:24.488922 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:24.488889    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f93d2fa-4486-4f9c-973e-7e0db7272176-maas-api-openshift-ca" pod "4f93d2fa-4486-4f9c-973e-7e0db7272176" (UID: "4f93d2fa-4486-4f9c-973e-7e0db7272176"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:34:24.490685 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:24.490659    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f93d2fa-4486-4f9c-973e-7e0db7272176-kube-api-access-5wzbs" pod "4f93d2fa-4486-4f9c-973e-7e0db7272176" (UID: "4f93d2fa-4486-4f9c-973e-7e0db7272176"). InnerVolumeSpecName "kube-api-access-5wzbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:34:24.490770 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:24.490659    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f93d2fa-4486-4f9c-973e-7e0db7272176-maas-api-xks-ca" pod "4f93d2fa-4486-4f9c-973e-7e0db7272176" (UID: "4f93d2fa-4486-4f9c-973e-7e0db7272176"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:34:24.589486 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:24.589458    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f93d2fa-4486-4f9c-973e-7e0db7272176-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:34:24.589486 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:24.589483    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wzbs\" (UniqueName: \"kubernetes.io/projected/4f93d2fa-4486-4f9c-973e-7e0db7272176-kube-api-access-5wzbs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:34:24.589626 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:24.589494    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f93d2fa-4486-4f9c-973e-7e0db7272176-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:34:25.187721 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:25.187687    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgw2k"
Nov 03 01:34:25.187902 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:25.187716    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgw2k" event={"ID":"4f93d2fa-4486-4f9c-973e-7e0db7272176","Type":"ContainerDied","Data":"a73eea18906ee538b18caea2459f484c28299b9eb90c1c5f16ff90f79a7ec53c"}
Nov 03 01:34:25.187902 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:25.187765    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a73eea18906ee538b18caea2459f484c28299b9eb90c1c5f16ff90f79a7ec53c"
Nov 03 01:34:27.025489 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:27.025451    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d68kq"]
Nov 03 01:34:27.035350 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:27.035320    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-d68kq"]
Nov 03 01:34:27.243294 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:27.243258    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8b8a3de-cc6b-489f-8a1d-8be74509e423" path="/var/lib/kubelet/pods/c8b8a3de-cc6b-489f-8a1d-8be74509e423/volumes"
Nov 03 01:34:34.690943 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:34.689998    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-cdvd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:34:34.690943 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:34.690076    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9" podUID="a125db3f-e711-4e19-ac5d-cbd3827be603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:34:34.849931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:34.848989    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hn6jd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:34:34.849931 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:34.849061    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd" podUID="99643392-d8d1-47c9-b9d4-e87e2c837f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:34:37.363384 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:34:37.363342    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:34:53.025575 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:53.025540    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bzl6r"]
Nov 03 01:34:53.035535 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:53.035507    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bzl6r"]
Nov 03 01:34:53.243078 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:34:53.243041    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e70b630-2c9d-49fe-89a1-f1249d7941f7" path="/var/lib/kubelet/pods/0e70b630-2c9d-49fe-89a1-f1249d7941f7/volumes"
Nov 03 01:35:07.362264 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:35:07.362215    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:35:18.959650 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:35:18.959609    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 01:35:18.964543 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:35:18.964514    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 01:35:18.969173 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:35:18.969151    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 01:35:18.973382 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:35:18.973366    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 01:35:22.306606 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:35:22.306569    5012 scope.go:122] "RemoveContainer" containerID="10348978cf354c0f8003c0d8e9c9acd5314ee93769c3370c317916ff17cb0d91"
Nov 03 01:35:22.316459 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:35:22.316437    5012 scope.go:122] "RemoveContainer" containerID="3e9cb645c81ac757ccf885f79f44518b66ad1c37ea3ae888f65981024e4ed8fd"
Nov 03 01:35:22.324960 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:35:22.324938    5012 scope.go:122] "RemoveContainer" containerID="9c36e28cdc158e1dcea20e7170eee05d6701355c9e10ce83fab8530730784d2b"
Nov 03 01:35:26.026150 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:35:26.026115    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-prg8l"]
Nov 03 01:35:26.035776 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:35:26.035733    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-prg8l"]
Nov 03 01:35:27.241994 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:35:27.241954    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="175f0a30-66ed-4f93-b5f9-32b956dfc6f0" path="/var/lib/kubelet/pods/175f0a30-66ed-4f93-b5f9-32b956dfc6f0/volumes"
Nov 03 01:35:37.362526 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:35:37.362490    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:35:39.031737 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:35:39.031699    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2sbm"]
Nov 03 01:35:39.041385 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:35:39.041356    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2sbm"]
Nov 03 01:35:39.242207 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:35:39.242174    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03873333-551f-4b0d-9ccd-fa856943acf0" path="/var/lib/kubelet/pods/03873333-551f-4b0d-9ccd-fa856943acf0/volumes"
Nov 03 01:36:04.695118 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:04.695019    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-cdvd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:36:04.696041 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:04.695996    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9" podUID="a125db3f-e711-4e19-ac5d-cbd3827be603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:36:04.837381 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:04.837342    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hn6jd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:36:04.837705 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:04.837673    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd" podUID="99643392-d8d1-47c9-b9d4-e87e2c837f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:36:06.286301 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:06.286268    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drgvb"]
Nov 03 01:36:06.286697 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:06.286667    5012 cpu_manager.go:414] "containerMap: removing container" podUID="4f93d2fa-4486-4f9c-973e-7e0db7272176" containerName="revoke-keys"
Nov 03 01:36:06.286697 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:06.286678    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f93d2fa-4486-4f9c-973e-7e0db7272176" containerName="revoke-keys"
Nov 03 01:36:06.286781 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:06.286761    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f93d2fa-4486-4f9c-973e-7e0db7272176" containerName="revoke-keys"
Nov 03 01:36:06.289983 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:06.289966    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drgvb"
Nov 03 01:36:06.315957 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:06.315932    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:36:06.316153 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:06.315994    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:36:06.316236 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:06.316018    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:36:06.316236 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:06.316068    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:36:06.319081 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:06.319057    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drgvb"]
Nov 03 01:36:06.378392 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:06.378360    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34f87501-6806-40c6-ba6f-51bc93763219-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-drgvb\" (UID: \"34f87501-6806-40c6-ba6f-51bc93763219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drgvb"
Nov 03 01:36:06.378598 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:06.378412    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34f87501-6806-40c6-ba6f-51bc93763219-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-drgvb\" (UID: \"34f87501-6806-40c6-ba6f-51bc93763219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drgvb"
Nov 03 01:36:06.378598 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:06.378528    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r9gx\" (UniqueName: \"kubernetes.io/projected/34f87501-6806-40c6-ba6f-51bc93763219-kube-api-access-7r9gx\") pod \"maas-api-revoke-sub-d576e68b-drgvb\" (UID: \"34f87501-6806-40c6-ba6f-51bc93763219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drgvb"
Nov 03 01:36:06.479785 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:06.479746    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34f87501-6806-40c6-ba6f-51bc93763219-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-drgvb\" (UID: \"34f87501-6806-40c6-ba6f-51bc93763219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drgvb"
Nov 03 01:36:06.479997 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:06.479799    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34f87501-6806-40c6-ba6f-51bc93763219-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-drgvb\" (UID: \"34f87501-6806-40c6-ba6f-51bc93763219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drgvb"
Nov 03 01:36:06.479997 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:06.479829    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r9gx\" (UniqueName: \"kubernetes.io/projected/34f87501-6806-40c6-ba6f-51bc93763219-kube-api-access-7r9gx\") pod \"maas-api-revoke-sub-d576e68b-drgvb\" (UID: \"34f87501-6806-40c6-ba6f-51bc93763219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drgvb"
Nov 03 01:36:06.480541 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:06.480516    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34f87501-6806-40c6-ba6f-51bc93763219-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-drgvb\" (UID: \"34f87501-6806-40c6-ba6f-51bc93763219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drgvb"
Nov 03 01:36:06.482150 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:06.482125    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34f87501-6806-40c6-ba6f-51bc93763219-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-drgvb\" (UID: \"34f87501-6806-40c6-ba6f-51bc93763219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drgvb"
Nov 03 01:36:06.496472 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:06.496447    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r9gx\" (UniqueName: \"kubernetes.io/projected/34f87501-6806-40c6-ba6f-51bc93763219-kube-api-access-7r9gx\") pod \"maas-api-revoke-sub-d576e68b-drgvb\" (UID: \"34f87501-6806-40c6-ba6f-51bc93763219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drgvb"
Nov 03 01:36:06.603549 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:06.603521    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drgvb"
Nov 03 01:36:06.897737 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:06.897705    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drgvb"]
Nov 03 01:36:06.899127 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:36:06.899099    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34f87501_6806_40c6_ba6f_51bc93763219.slice/crio-e67156b5468e75bb2b8cd81d7bab76357d02de999624df09826848ef802e20bb WatchSource:0}: Error finding container e67156b5468e75bb2b8cd81d7bab76357d02de999624df09826848ef802e20bb: Status 404 returned error can't find the container with id e67156b5468e75bb2b8cd81d7bab76357d02de999624df09826848ef802e20bb
Nov 03 01:36:07.362355 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:36:07.362319    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:36:07.891231 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:07.891190    5012 generic.go:356] "Generic (PLEG): container finished" podID="34f87501-6806-40c6-ba6f-51bc93763219" containerID="ccff65683816d9fa2fd5f257c8f108b4ac69d33fc6b488ccb6a33096378fe7b3" exitCode=0
Nov 03 01:36:07.891419 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:07.891269    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drgvb" event={"ID":"34f87501-6806-40c6-ba6f-51bc93763219","Type":"ContainerDied","Data":"ccff65683816d9fa2fd5f257c8f108b4ac69d33fc6b488ccb6a33096378fe7b3"}
Nov 03 01:36:07.891419 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:07.891321    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drgvb" event={"ID":"34f87501-6806-40c6-ba6f-51bc93763219","Type":"ContainerStarted","Data":"e67156b5468e75bb2b8cd81d7bab76357d02de999624df09826848ef802e20bb"}
Nov 03 01:36:09.025727 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:09.025704    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drgvb"
Nov 03 01:36:09.104812 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:09.104787    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34f87501-6806-40c6-ba6f-51bc93763219-kube-api-access-7r9gx\" (UniqueName: \"kubernetes.io/projected/34f87501-6806-40c6-ba6f-51bc93763219-kube-api-access-7r9gx\") pod \"34f87501-6806-40c6-ba6f-51bc93763219\" (UID: \"34f87501-6806-40c6-ba6f-51bc93763219\") "
Nov 03 01:36:09.104983 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:09.104846    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34f87501-6806-40c6-ba6f-51bc93763219-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34f87501-6806-40c6-ba6f-51bc93763219-maas-api-openshift-ca\") pod \"34f87501-6806-40c6-ba6f-51bc93763219\" (UID: \"34f87501-6806-40c6-ba6f-51bc93763219\") "
Nov 03 01:36:09.104983 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:09.104944    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34f87501-6806-40c6-ba6f-51bc93763219-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34f87501-6806-40c6-ba6f-51bc93763219-maas-api-xks-ca\") pod \"34f87501-6806-40c6-ba6f-51bc93763219\" (UID: \"34f87501-6806-40c6-ba6f-51bc93763219\") "
Nov 03 01:36:09.105205 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:09.105177    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34f87501-6806-40c6-ba6f-51bc93763219-maas-api-openshift-ca" pod "34f87501-6806-40c6-ba6f-51bc93763219" (UID: "34f87501-6806-40c6-ba6f-51bc93763219"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:36:09.106974 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:09.106951    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34f87501-6806-40c6-ba6f-51bc93763219-maas-api-xks-ca" pod "34f87501-6806-40c6-ba6f-51bc93763219" (UID: "34f87501-6806-40c6-ba6f-51bc93763219"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:36:09.106974 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:09.106961    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f87501-6806-40c6-ba6f-51bc93763219-kube-api-access-7r9gx" pod "34f87501-6806-40c6-ba6f-51bc93763219" (UID: "34f87501-6806-40c6-ba6f-51bc93763219"). InnerVolumeSpecName "kube-api-access-7r9gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:36:09.205609 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:09.205549    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34f87501-6806-40c6-ba6f-51bc93763219-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:36:09.205609 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:09.205576    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r9gx\" (UniqueName: \"kubernetes.io/projected/34f87501-6806-40c6-ba6f-51bc93763219-kube-api-access-7r9gx\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:36:09.205609 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:09.205588    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34f87501-6806-40c6-ba6f-51bc93763219-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:36:09.905384 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:09.905355    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drgvb"
Nov 03 01:36:09.905384 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:09.905377    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drgvb" event={"ID":"34f87501-6806-40c6-ba6f-51bc93763219","Type":"ContainerDied","Data":"e67156b5468e75bb2b8cd81d7bab76357d02de999624df09826848ef802e20bb"}
Nov 03 01:36:09.905616 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:09.905419    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e67156b5468e75bb2b8cd81d7bab76357d02de999624df09826848ef802e20bb"
Nov 03 01:36:21.354938 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:21.352249    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k5stq"]
Nov 03 01:36:21.354938 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:21.352925    5012 cpu_manager.go:414] "containerMap: removing container" podUID="34f87501-6806-40c6-ba6f-51bc93763219" containerName="revoke-keys"
Nov 03 01:36:21.354938 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:21.352944    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34f87501-6806-40c6-ba6f-51bc93763219" containerName="revoke-keys"
Nov 03 01:36:21.354938 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:21.353068    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="34f87501-6806-40c6-ba6f-51bc93763219" containerName="revoke-keys"
Nov 03 01:36:21.426933 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:21.426330    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k5stq"]
Nov 03 01:36:21.426933 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:21.426467    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k5stq"
Nov 03 01:36:21.475277 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:21.475243    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:36:21.475615 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:21.475594    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:36:21.475766 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:21.475752    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:36:21.475999 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:21.475981    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:36:21.530537 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:21.530489    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23689016-fe12-41e3-8740-efcce91800b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-k5stq\" (UID: \"23689016-fe12-41e3-8740-efcce91800b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k5stq"
Nov 03 01:36:21.530861 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:21.530839    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r54hj\" (UniqueName: \"kubernetes.io/projected/23689016-fe12-41e3-8740-efcce91800b1-kube-api-access-r54hj\") pod \"maas-api-revoke-sub-d2bc93bc-k5stq\" (UID: \"23689016-fe12-41e3-8740-efcce91800b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k5stq"
Nov 03 01:36:21.531053 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:21.531032    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23689016-fe12-41e3-8740-efcce91800b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-k5stq\" (UID: \"23689016-fe12-41e3-8740-efcce91800b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k5stq"
Nov 03 01:36:21.643507 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:21.643405    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23689016-fe12-41e3-8740-efcce91800b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-k5stq\" (UID: \"23689016-fe12-41e3-8740-efcce91800b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k5stq"
Nov 03 01:36:21.644372 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:21.644344    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23689016-fe12-41e3-8740-efcce91800b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-k5stq\" (UID: \"23689016-fe12-41e3-8740-efcce91800b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k5stq"
Nov 03 01:36:21.644619 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:21.644596    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r54hj\" (UniqueName: \"kubernetes.io/projected/23689016-fe12-41e3-8740-efcce91800b1-kube-api-access-r54hj\") pod \"maas-api-revoke-sub-d2bc93bc-k5stq\" (UID: \"23689016-fe12-41e3-8740-efcce91800b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k5stq"
Nov 03 01:36:21.660935 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:21.645821    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23689016-fe12-41e3-8740-efcce91800b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-k5stq\" (UID: \"23689016-fe12-41e3-8740-efcce91800b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k5stq"
Nov 03 01:36:21.661657 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:21.654513    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23689016-fe12-41e3-8740-efcce91800b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-k5stq\" (UID: \"23689016-fe12-41e3-8740-efcce91800b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k5stq"
Nov 03 01:36:21.664724 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:21.664698    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r54hj\" (UniqueName: \"kubernetes.io/projected/23689016-fe12-41e3-8740-efcce91800b1-kube-api-access-r54hj\") pod \"maas-api-revoke-sub-d2bc93bc-k5stq\" (UID: \"23689016-fe12-41e3-8740-efcce91800b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k5stq"
Nov 03 01:36:21.764874 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:21.764838    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k5stq"
Nov 03 01:36:22.253990 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:22.217162    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k5stq"]
Nov 03 01:36:22.259285 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:36:22.259235    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23689016_fe12_41e3_8740_efcce91800b1.slice/crio-807b74f6def37cf4aa88ad3e8bcbce999e8fbccb7cbf8fe4fa645ce8c331949f WatchSource:0}: Error finding container 807b74f6def37cf4aa88ad3e8bcbce999e8fbccb7cbf8fe4fa645ce8c331949f: Status 404 returned error can't find the container with id 807b74f6def37cf4aa88ad3e8bcbce999e8fbccb7cbf8fe4fa645ce8c331949f
Nov 03 01:36:22.368930 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:22.368871    5012 scope.go:122] "RemoveContainer" containerID="dfba3335cf69a2ede98dd7e0b45eeb8d79bc232365e610378080e78b8deeed2b"
Nov 03 01:36:22.378046 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:22.378020    5012 scope.go:122] "RemoveContainer" containerID="1c80bf1eb09ba9192af988785b6786d02d5a9f6d2439a13b73d31484e540abaa"
Nov 03 01:36:23.079316 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:23.079280    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k5stq" event={"ID":"23689016-fe12-41e3-8740-efcce91800b1","Type":"ContainerStarted","Data":"3b7dda6a2ee400d1ac309c00f77e3bd827df815141b5d309d6c506c8e786cb23"}
Nov 03 01:36:23.079487 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:23.079344    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k5stq" event={"ID":"23689016-fe12-41e3-8740-efcce91800b1","Type":"ContainerStarted","Data":"807b74f6def37cf4aa88ad3e8bcbce999e8fbccb7cbf8fe4fa645ce8c331949f"}
Nov 03 01:36:23.120056 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:23.119301    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k5stq" podStartSLOduration=2.119282278 podStartE2EDuration="2.119282278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:36:23.114223187 +0000 UTC m=+5465.993807232" watchObservedRunningTime="2026-11-03 01:36:23.119282278 +0000 UTC m=+5465.998866322"
Nov 03 01:36:24.087105 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:24.087067    5012 generic.go:356] "Generic (PLEG): container finished" podID="23689016-fe12-41e3-8740-efcce91800b1" containerID="3b7dda6a2ee400d1ac309c00f77e3bd827df815141b5d309d6c506c8e786cb23" exitCode=0
Nov 03 01:36:24.087463 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:24.087139    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k5stq" event={"ID":"23689016-fe12-41e3-8740-efcce91800b1","Type":"ContainerDied","Data":"3b7dda6a2ee400d1ac309c00f77e3bd827df815141b5d309d6c506c8e786cb23"}
Nov 03 01:36:25.237581 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:25.237554    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k5stq"
Nov 03 01:36:25.289216 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:25.289188    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23689016-fe12-41e3-8740-efcce91800b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23689016-fe12-41e3-8740-efcce91800b1-maas-api-xks-ca\") pod \"23689016-fe12-41e3-8740-efcce91800b1\" (UID: \"23689016-fe12-41e3-8740-efcce91800b1\") "
Nov 03 01:36:25.289354 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:25.289244    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23689016-fe12-41e3-8740-efcce91800b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23689016-fe12-41e3-8740-efcce91800b1-maas-api-openshift-ca\") pod \"23689016-fe12-41e3-8740-efcce91800b1\" (UID: \"23689016-fe12-41e3-8740-efcce91800b1\") "
Nov 03 01:36:25.289354 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:25.289283    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23689016-fe12-41e3-8740-efcce91800b1-kube-api-access-r54hj\" (UniqueName: \"kubernetes.io/projected/23689016-fe12-41e3-8740-efcce91800b1-kube-api-access-r54hj\") pod \"23689016-fe12-41e3-8740-efcce91800b1\" (UID: \"23689016-fe12-41e3-8740-efcce91800b1\") "
Nov 03 01:36:25.289713 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:25.289681    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23689016-fe12-41e3-8740-efcce91800b1-maas-api-openshift-ca" pod "23689016-fe12-41e3-8740-efcce91800b1" (UID: "23689016-fe12-41e3-8740-efcce91800b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:36:25.291471 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:25.291444    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23689016-fe12-41e3-8740-efcce91800b1-kube-api-access-r54hj" pod "23689016-fe12-41e3-8740-efcce91800b1" (UID: "23689016-fe12-41e3-8740-efcce91800b1"). InnerVolumeSpecName "kube-api-access-r54hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:36:25.291471 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:25.291454    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23689016-fe12-41e3-8740-efcce91800b1-maas-api-xks-ca" pod "23689016-fe12-41e3-8740-efcce91800b1" (UID: "23689016-fe12-41e3-8740-efcce91800b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:36:25.390623 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:25.390594    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23689016-fe12-41e3-8740-efcce91800b1-maas-api-xks-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:36:25.390623 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:25.390620    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23689016-fe12-41e3-8740-efcce91800b1-maas-api-openshift-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:36:25.390793 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:25.390630    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r54hj\" (UniqueName: \"kubernetes.io/projected/23689016-fe12-41e3-8740-efcce91800b1-kube-api-access-r54hj\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:36:26.101346 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:26.101309    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k5stq"
Nov 03 01:36:26.101346 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:26.101337    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k5stq" event={"ID":"23689016-fe12-41e3-8740-efcce91800b1","Type":"ContainerDied","Data":"807b74f6def37cf4aa88ad3e8bcbce999e8fbccb7cbf8fe4fa645ce8c331949f"}
Nov 03 01:36:26.101586 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:26.101386    5012 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="807b74f6def37cf4aa88ad3e8bcbce999e8fbccb7cbf8fe4fa645ce8c331949f"
Nov 03 01:36:33.475897 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:33.475862    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6675486f96-8g5bj"]
Nov 03 01:36:33.476344 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:33.476322    5012 cpu_manager.go:414] "containerMap: removing container" podUID="23689016-fe12-41e3-8740-efcce91800b1" containerName="revoke-keys"
Nov 03 01:36:33.476393 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:33.476346    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23689016-fe12-41e3-8740-efcce91800b1" containerName="revoke-keys"
Nov 03 01:36:33.476488 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:33.476465    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="23689016-fe12-41e3-8740-efcce91800b1" containerName="revoke-keys"
Nov 03 01:36:33.479716 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:33.479696    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6675486f96-8g5bj"
Nov 03 01:36:33.505629 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:33.505603    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:36:33.505786 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:33.505677    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:36:33.505786 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:33.505723    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 01:36:33.506082 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:33.506064    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-x7pjw\""
Nov 03 01:36:33.509367 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:33.509345    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6675486f96-8g5bj"]
Nov 03 01:36:33.516407 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:33.516386    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 01:36:33.576657 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:33.576626    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqlkl\" (UniqueName: \"kubernetes.io/projected/db3c4862-4118-42a3-bb11-dbc1d638f12f-kube-api-access-cqlkl\") pod \"maas-api-6675486f96-8g5bj\" (UID: \"db3c4862-4118-42a3-bb11-dbc1d638f12f\") " pod="odh-ai-gateway-infra/maas-api-6675486f96-8g5bj"
Nov 03 01:36:33.576768 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:33.576665    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db3c4862-4118-42a3-bb11-dbc1d638f12f-maas-api-tls\") pod \"maas-api-6675486f96-8g5bj\" (UID: \"db3c4862-4118-42a3-bb11-dbc1d638f12f\") " pod="odh-ai-gateway-infra/maas-api-6675486f96-8g5bj"
Nov 03 01:36:33.576825 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:33.576800    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db3c4862-4118-42a3-bb11-dbc1d638f12f-metrics-certs\") pod \"maas-api-6675486f96-8g5bj\" (UID: \"db3c4862-4118-42a3-bb11-dbc1d638f12f\") " pod="odh-ai-gateway-infra/maas-api-6675486f96-8g5bj"
Nov 03 01:36:33.576905 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:33.576890    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db3c4862-4118-42a3-bb11-dbc1d638f12f-openshift-service-ca\") pod \"maas-api-6675486f96-8g5bj\" (UID: \"db3c4862-4118-42a3-bb11-dbc1d638f12f\") " pod="odh-ai-gateway-infra/maas-api-6675486f96-8g5bj"
Nov 03 01:36:33.677614 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:33.677586    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db3c4862-4118-42a3-bb11-dbc1d638f12f-openshift-service-ca\") pod \"maas-api-6675486f96-8g5bj\" (UID: \"db3c4862-4118-42a3-bb11-dbc1d638f12f\") " pod="odh-ai-gateway-infra/maas-api-6675486f96-8g5bj"
Nov 03 01:36:33.677741 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:33.677629    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqlkl\" (UniqueName: \"kubernetes.io/projected/db3c4862-4118-42a3-bb11-dbc1d638f12f-kube-api-access-cqlkl\") pod \"maas-api-6675486f96-8g5bj\" (UID: \"db3c4862-4118-42a3-bb11-dbc1d638f12f\") " pod="odh-ai-gateway-infra/maas-api-6675486f96-8g5bj"
Nov 03 01:36:33.677741 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:33.677660    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db3c4862-4118-42a3-bb11-dbc1d638f12f-maas-api-tls\") pod \"maas-api-6675486f96-8g5bj\" (UID: \"db3c4862-4118-42a3-bb11-dbc1d638f12f\") " pod="odh-ai-gateway-infra/maas-api-6675486f96-8g5bj"
Nov 03 01:36:33.677741 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:33.677721    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db3c4862-4118-42a3-bb11-dbc1d638f12f-metrics-certs\") pod \"maas-api-6675486f96-8g5bj\" (UID: \"db3c4862-4118-42a3-bb11-dbc1d638f12f\") " pod="odh-ai-gateway-infra/maas-api-6675486f96-8g5bj"
Nov 03 01:36:33.678391 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:33.678370    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db3c4862-4118-42a3-bb11-dbc1d638f12f-openshift-service-ca\") pod \"maas-api-6675486f96-8g5bj\" (UID: \"db3c4862-4118-42a3-bb11-dbc1d638f12f\") " pod="odh-ai-gateway-infra/maas-api-6675486f96-8g5bj"
Nov 03 01:36:33.680073 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:33.680052    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db3c4862-4118-42a3-bb11-dbc1d638f12f-metrics-certs\") pod \"maas-api-6675486f96-8g5bj\" (UID: \"db3c4862-4118-42a3-bb11-dbc1d638f12f\") " pod="odh-ai-gateway-infra/maas-api-6675486f96-8g5bj"
Nov 03 01:36:33.680224 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:33.680207    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db3c4862-4118-42a3-bb11-dbc1d638f12f-maas-api-tls\") pod \"maas-api-6675486f96-8g5bj\" (UID: \"db3c4862-4118-42a3-bb11-dbc1d638f12f\") " pod="odh-ai-gateway-infra/maas-api-6675486f96-8g5bj"
Nov 03 01:36:33.694024 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:33.694001    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqlkl\" (UniqueName: \"kubernetes.io/projected/db3c4862-4118-42a3-bb11-dbc1d638f12f-kube-api-access-cqlkl\") pod \"maas-api-6675486f96-8g5bj\" (UID: \"db3c4862-4118-42a3-bb11-dbc1d638f12f\") " pod="odh-ai-gateway-infra/maas-api-6675486f96-8g5bj"
Nov 03 01:36:33.794047 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:33.793963    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6675486f96-8g5bj"
Nov 03 01:36:34.088825 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:34.088792    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6675486f96-8g5bj"]
Nov 03 01:36:34.090039 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:36:34.090011    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb3c4862_4118_42a3_bb11_dbc1d638f12f.slice/crio-211c1c682531e999cb3d7e2a0bd9089c0f6b0d9e9afc37fc89fbb6ba825cce89 WatchSource:0}: Error finding container 211c1c682531e999cb3d7e2a0bd9089c0f6b0d9e9afc37fc89fbb6ba825cce89: Status 404 returned error can't find the container with id 211c1c682531e999cb3d7e2a0bd9089c0f6b0d9e9afc37fc89fbb6ba825cce89
Nov 03 01:36:34.091276 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:34.091255    5012 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 01:36:34.161770 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:34.161730    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6675486f96-8g5bj" event={"ID":"db3c4862-4118-42a3-bb11-dbc1d638f12f","Type":"ContainerStarted","Data":"211c1c682531e999cb3d7e2a0bd9089c0f6b0d9e9afc37fc89fbb6ba825cce89"}
Nov 03 01:36:35.170501 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:35.170457    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6675486f96-8g5bj" event={"ID":"db3c4862-4118-42a3-bb11-dbc1d638f12f","Type":"ContainerStarted","Data":"a25bf782e7e875d65f695f547dba6398ccd9844d02203765a6133d696a2a0038"}
Nov 03 01:36:35.171005 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:35.170549    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6675486f96-8g5bj"
Nov 03 01:36:35.207256 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:35.207183    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6675486f96-8g5bj" podStartSLOduration=1.869901979 podStartE2EDuration="2.207165907s" totalImagesPullingTime="337.263928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:36:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:36:35.204301653 +0000 UTC m=+5478.083885696" watchObservedRunningTime="2026-11-03 01:36:35.207165907 +0000 UTC m=+5478.086749950"
Nov 03 01:36:37.036188 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:37.036153    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-77dld"]
Nov 03 01:36:37.046552 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:37.046505    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-77dld"]
Nov 03 01:36:37.251303 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:37.251265    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3da4e286-2604-498d-91f1-8c88d350eb4a" path="/var/lib/kubelet/pods/3da4e286-2604-498d-91f1-8c88d350eb4a/volumes"
Nov 03 01:36:37.364614 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:36:37.364572    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:36:40.851838 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:40.851797    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6675486f96-8g5bj"]
Nov 03 01:36:40.852577 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:40.852061    5012 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6675486f96-8g5bj" podUID="db3c4862-4118-42a3-bb11-dbc1d638f12f" containerName="maas-api" containerID="cri-o://a25bf782e7e875d65f695f547dba6398ccd9844d02203765a6133d696a2a0038" gracePeriod=30
Nov 03 01:36:40.858416 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:40.858392    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6675486f96-8g5bj"
Nov 03 01:36:40.882644 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:40.882615    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6cdc687864-vbj7c"]
Nov 03 01:36:40.886363 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:40.886344    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cdc687864-vbj7c"
Nov 03 01:36:40.906319 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:40.906292    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cdc687864-vbj7c"]
Nov 03 01:36:40.949145 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:40.949109    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz557\" (UniqueName: \"kubernetes.io/projected/a13f8b66-2eca-4a75-a954-5ea44360980d-kube-api-access-gz557\") pod \"maas-api-6cdc687864-vbj7c\" (UID: \"a13f8b66-2eca-4a75-a954-5ea44360980d\") " pod="odh-ai-gateway-infra/maas-api-6cdc687864-vbj7c"
Nov 03 01:36:40.949355 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:40.949193    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a13f8b66-2eca-4a75-a954-5ea44360980d-openshift-service-ca\") pod \"maas-api-6cdc687864-vbj7c\" (UID: \"a13f8b66-2eca-4a75-a954-5ea44360980d\") " pod="odh-ai-gateway-infra/maas-api-6cdc687864-vbj7c"
Nov 03 01:36:40.949355 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:40.949318    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a13f8b66-2eca-4a75-a954-5ea44360980d-maas-api-tls\") pod \"maas-api-6cdc687864-vbj7c\" (UID: \"a13f8b66-2eca-4a75-a954-5ea44360980d\") " pod="odh-ai-gateway-infra/maas-api-6cdc687864-vbj7c"
Nov 03 01:36:40.949487 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:40.949359    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a13f8b66-2eca-4a75-a954-5ea44360980d-metrics-certs\") pod \"maas-api-6cdc687864-vbj7c\" (UID: \"a13f8b66-2eca-4a75-a954-5ea44360980d\") " pod="odh-ai-gateway-infra/maas-api-6cdc687864-vbj7c"
Nov 03 01:36:41.050739 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.050687    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a13f8b66-2eca-4a75-a954-5ea44360980d-metrics-certs\") pod \"maas-api-6cdc687864-vbj7c\" (UID: \"a13f8b66-2eca-4a75-a954-5ea44360980d\") " pod="odh-ai-gateway-infra/maas-api-6cdc687864-vbj7c"
Nov 03 01:36:41.051466 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.050796    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz557\" (UniqueName: \"kubernetes.io/projected/a13f8b66-2eca-4a75-a954-5ea44360980d-kube-api-access-gz557\") pod \"maas-api-6cdc687864-vbj7c\" (UID: \"a13f8b66-2eca-4a75-a954-5ea44360980d\") " pod="odh-ai-gateway-infra/maas-api-6cdc687864-vbj7c"
Nov 03 01:36:41.051466 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.050894    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a13f8b66-2eca-4a75-a954-5ea44360980d-openshift-service-ca\") pod \"maas-api-6cdc687864-vbj7c\" (UID: \"a13f8b66-2eca-4a75-a954-5ea44360980d\") " pod="odh-ai-gateway-infra/maas-api-6cdc687864-vbj7c"
Nov 03 01:36:41.051466 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.051011    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a13f8b66-2eca-4a75-a954-5ea44360980d-maas-api-tls\") pod \"maas-api-6cdc687864-vbj7c\" (UID: \"a13f8b66-2eca-4a75-a954-5ea44360980d\") " pod="odh-ai-gateway-infra/maas-api-6cdc687864-vbj7c"
Nov 03 01:36:41.052029 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.052003    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a13f8b66-2eca-4a75-a954-5ea44360980d-openshift-service-ca\") pod \"maas-api-6cdc687864-vbj7c\" (UID: \"a13f8b66-2eca-4a75-a954-5ea44360980d\") " pod="odh-ai-gateway-infra/maas-api-6cdc687864-vbj7c"
Nov 03 01:36:41.053968 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.053939    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a13f8b66-2eca-4a75-a954-5ea44360980d-maas-api-tls\") pod \"maas-api-6cdc687864-vbj7c\" (UID: \"a13f8b66-2eca-4a75-a954-5ea44360980d\") " pod="odh-ai-gateway-infra/maas-api-6cdc687864-vbj7c"
Nov 03 01:36:41.054125 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.054099    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a13f8b66-2eca-4a75-a954-5ea44360980d-metrics-certs\") pod \"maas-api-6cdc687864-vbj7c\" (UID: \"a13f8b66-2eca-4a75-a954-5ea44360980d\") " pod="odh-ai-gateway-infra/maas-api-6cdc687864-vbj7c"
Nov 03 01:36:41.068814 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.068782    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz557\" (UniqueName: \"kubernetes.io/projected/a13f8b66-2eca-4a75-a954-5ea44360980d-kube-api-access-gz557\") pod \"maas-api-6cdc687864-vbj7c\" (UID: \"a13f8b66-2eca-4a75-a954-5ea44360980d\") " pod="odh-ai-gateway-infra/maas-api-6cdc687864-vbj7c"
Nov 03 01:36:41.116809 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.116777    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6675486f96-8g5bj"
Nov 03 01:36:41.210358 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.210326    5012 generic.go:356] "Generic (PLEG): container finished" podID="db3c4862-4118-42a3-bb11-dbc1d638f12f" containerID="a25bf782e7e875d65f695f547dba6398ccd9844d02203765a6133d696a2a0038" exitCode=0
Nov 03 01:36:41.210558 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.210381    5012 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6675486f96-8g5bj"
Nov 03 01:36:41.210558 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.210427    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6675486f96-8g5bj" event={"ID":"db3c4862-4118-42a3-bb11-dbc1d638f12f","Type":"ContainerDied","Data":"a25bf782e7e875d65f695f547dba6398ccd9844d02203765a6133d696a2a0038"}
Nov 03 01:36:41.210558 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.210490    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6675486f96-8g5bj" event={"ID":"db3c4862-4118-42a3-bb11-dbc1d638f12f","Type":"ContainerDied","Data":"211c1c682531e999cb3d7e2a0bd9089c0f6b0d9e9afc37fc89fbb6ba825cce89"}
Nov 03 01:36:41.210558 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.210513    5012 scope.go:122] "RemoveContainer" containerID="a25bf782e7e875d65f695f547dba6398ccd9844d02203765a6133d696a2a0038"
Nov 03 01:36:41.224158 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.224126    5012 scope.go:122] "RemoveContainer" containerID="a25bf782e7e875d65f695f547dba6398ccd9844d02203765a6133d696a2a0038"
Nov 03 01:36:41.224511 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:36:41.224483    5012 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a25bf782e7e875d65f695f547dba6398ccd9844d02203765a6133d696a2a0038\": container with ID starting with a25bf782e7e875d65f695f547dba6398ccd9844d02203765a6133d696a2a0038 not found: ID does not exist" containerID="a25bf782e7e875d65f695f547dba6398ccd9844d02203765a6133d696a2a0038"
Nov 03 01:36:41.224568 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.224524    5012 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25bf782e7e875d65f695f547dba6398ccd9844d02203765a6133d696a2a0038"} err="failed to get container status \"a25bf782e7e875d65f695f547dba6398ccd9844d02203765a6133d696a2a0038\": rpc error: code = NotFound desc = could not find container \"a25bf782e7e875d65f695f547dba6398ccd9844d02203765a6133d696a2a0038\": container with ID starting with a25bf782e7e875d65f695f547dba6398ccd9844d02203765a6133d696a2a0038 not found: ID does not exist"
Nov 03 01:36:41.252306 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.252265    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db3c4862-4118-42a3-bb11-dbc1d638f12f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/db3c4862-4118-42a3-bb11-dbc1d638f12f-metrics-certs\") pod \"db3c4862-4118-42a3-bb11-dbc1d638f12f\" (UID: \"db3c4862-4118-42a3-bb11-dbc1d638f12f\") "
Nov 03 01:36:41.252306 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.252316    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db3c4862-4118-42a3-bb11-dbc1d638f12f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db3c4862-4118-42a3-bb11-dbc1d638f12f-openshift-service-ca\") pod \"db3c4862-4118-42a3-bb11-dbc1d638f12f\" (UID: \"db3c4862-4118-42a3-bb11-dbc1d638f12f\") "
Nov 03 01:36:41.252550 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.252407    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db3c4862-4118-42a3-bb11-dbc1d638f12f-kube-api-access-cqlkl\" (UniqueName: \"kubernetes.io/projected/db3c4862-4118-42a3-bb11-dbc1d638f12f-kube-api-access-cqlkl\") pod \"db3c4862-4118-42a3-bb11-dbc1d638f12f\" (UID: \"db3c4862-4118-42a3-bb11-dbc1d638f12f\") "
Nov 03 01:36:41.252550 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.252449    5012 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db3c4862-4118-42a3-bb11-dbc1d638f12f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db3c4862-4118-42a3-bb11-dbc1d638f12f-maas-api-tls\") pod \"db3c4862-4118-42a3-bb11-dbc1d638f12f\" (UID: \"db3c4862-4118-42a3-bb11-dbc1d638f12f\") "
Nov 03 01:36:41.252766 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.252732    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db3c4862-4118-42a3-bb11-dbc1d638f12f-openshift-service-ca" pod "db3c4862-4118-42a3-bb11-dbc1d638f12f" (UID: "db3c4862-4118-42a3-bb11-dbc1d638f12f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:36:41.254549 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.254517    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db3c4862-4118-42a3-bb11-dbc1d638f12f-metrics-certs" pod "db3c4862-4118-42a3-bb11-dbc1d638f12f" (UID: "db3c4862-4118-42a3-bb11-dbc1d638f12f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:36:41.254663 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.254573    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db3c4862-4118-42a3-bb11-dbc1d638f12f-maas-api-tls" pod "db3c4862-4118-42a3-bb11-dbc1d638f12f" (UID: "db3c4862-4118-42a3-bb11-dbc1d638f12f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:36:41.254663 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.254575    5012 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3c4862-4118-42a3-bb11-dbc1d638f12f-kube-api-access-cqlkl" pod "db3c4862-4118-42a3-bb11-dbc1d638f12f" (UID: "db3c4862-4118-42a3-bb11-dbc1d638f12f"). InnerVolumeSpecName "kube-api-access-cqlkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:36:41.271188 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.271143    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cdc687864-vbj7c"
Nov 03 01:36:41.354313 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.354276    5012 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqlkl\" (UniqueName: \"kubernetes.io/projected/db3c4862-4118-42a3-bb11-dbc1d638f12f-kube-api-access-cqlkl\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:36:41.354313 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.354313    5012 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db3c4862-4118-42a3-bb11-dbc1d638f12f-maas-api-tls\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:36:41.354598 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.354328    5012 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db3c4862-4118-42a3-bb11-dbc1d638f12f-metrics-certs\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:36:41.354598 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.354341    5012 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db3c4862-4118-42a3-bb11-dbc1d638f12f-openshift-service-ca\") on node \"ip-10-0-133-161.ec2.internal\" DevicePath \"\""
Nov 03 01:36:41.554162 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.554130    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6675486f96-8g5bj"]
Nov 03 01:36:41.564477 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.564450    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6675486f96-8g5bj"]
Nov 03 01:36:41.568294 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:41.568165    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cdc687864-vbj7c"]
Nov 03 01:36:41.571232 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:36:41.571197    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13f8b66_2eca_4a75_a954_5ea44360980d.slice/crio-288e3d4c1e5395bc0c5a3abf887bf22718d566cfb0dae457013e0e0b10150d1c WatchSource:0}: Error finding container 288e3d4c1e5395bc0c5a3abf887bf22718d566cfb0dae457013e0e0b10150d1c: Status 404 returned error can't find the container with id 288e3d4c1e5395bc0c5a3abf887bf22718d566cfb0dae457013e0e0b10150d1c
Nov 03 01:36:42.218943 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:42.218842    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cdc687864-vbj7c" event={"ID":"a13f8b66-2eca-4a75-a954-5ea44360980d","Type":"ContainerStarted","Data":"61c0899dbbfbe2975283b53a376a343ae6617d5aeab82f1c1e5841d1aae2568c"}
Nov 03 01:36:42.219314 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:42.218949    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cdc687864-vbj7c" event={"ID":"a13f8b66-2eca-4a75-a954-5ea44360980d","Type":"ContainerStarted","Data":"288e3d4c1e5395bc0c5a3abf887bf22718d566cfb0dae457013e0e0b10150d1c"}
Nov 03 01:36:42.219314 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:42.219087    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6cdc687864-vbj7c"
Nov 03 01:36:42.257132 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:42.257084    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6cdc687864-vbj7c" podStartSLOduration=1.958227794 podStartE2EDuration="2.257070527s" totalImagesPullingTime="298.842733ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:36:42.254382531 +0000 UTC m=+5485.133966573" watchObservedRunningTime="2026-11-03 01:36:42.257070527 +0000 UTC m=+5485.136654593"
Nov 03 01:36:43.242885 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:43.242846    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db3c4862-4118-42a3-bb11-dbc1d638f12f" path="/var/lib/kubelet/pods/db3c4862-4118-42a3-bb11-dbc1d638f12f/volumes"
Nov 03 01:36:48.026353 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:48.026307    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zn96c"]
Nov 03 01:36:48.036715 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:48.036682    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-zn96c"]
Nov 03 01:36:48.231845 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:48.231812    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6cdc687864-vbj7c"
Nov 03 01:36:49.245446 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:36:49.245394    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="762ca9fc-fe6e-456e-abca-2413bd52896c" path="/var/lib/kubelet/pods/762ca9fc-fe6e-456e-abca-2413bd52896c/volumes"
Nov 03 01:37:07.361772 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:37:07.361732    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:37:22.473861 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:37:22.473824    5012 scope.go:122] "RemoveContainer" containerID="4fb8ef7be238647aeb85b67e990b07d243a66b2293c67952a2713d860c90d036"
Nov 03 01:37:22.483402 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:37:22.483381    5012 scope.go:122] "RemoveContainer" containerID="3b939bf914743854c71b0df0d5bcabf93168c3f8ce91dcfd1a20e91e0739ac2c"
Nov 03 01:37:22.491958 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:37:22.491939    5012 scope.go:122] "RemoveContainer" containerID="4ec8004b70b4f760d4e8f70fb2e4dd68da46926f332d12bbc69c1cc20e3fad54"
Nov 03 01:37:22.500134 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:37:22.500117    5012 scope.go:122] "RemoveContainer" containerID="ff1742cd52547c37b07be04acf611cd997e3f9b3818f5dcd73d0897bd18cc2b9"
Nov 03 01:37:37.362738 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:37:37.362701    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:38:07.361878 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:38:07.361835    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:38:09.032171 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:38:09.032082    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75559f53-csjhd"]
Nov 03 01:38:09.042343 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:38:09.042312    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75559f53-csjhd"]
Nov 03 01:38:09.242625 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:38:09.242592    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a98939a9-969d-41c3-99c5-6526d48f2f08" path="/var/lib/kubelet/pods/a98939a9-969d-41c3-99c5-6526d48f2f08/volumes"
Nov 03 01:38:22.540001 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:38:22.539962    5012 scope.go:122] "RemoveContainer" containerID="d7736154527bbbdd1704ae210d4d74690c6daf9f76ad40678d198190a2289bd2"
Nov 03 01:38:22.549899 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:38:22.549877    5012 scope.go:122] "RemoveContainer" containerID="51d14475d509f5491ac5efaf5f3291001ed81e81eea8c4345b13bd9734f213bf"
Nov 03 01:38:22.559194 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:38:22.559173    5012 scope.go:122] "RemoveContainer" containerID="886ad9eea6a9f8d3bf31f5fd86125899f9a696abe47121d2fe57f70b19febe68"
Nov 03 01:38:37.378365 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:38:37.378311    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:39:07.361725 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:39:07.361637    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:39:22.592167 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:39:22.592126    5012 scope.go:122] "RemoveContainer" containerID="e68984f592263199fcdd01234941949937e75b9fcd98e3fa5b0b15e94747ecd3"
Nov 03 01:39:25.042036 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:39:25.041969    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgw2k"]
Nov 03 01:39:25.055953 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:39:25.055496    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kgw2k"]
Nov 03 01:39:25.263815 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:39:25.263776    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f93d2fa-4486-4f9c-973e-7e0db7272176" path="/var/lib/kubelet/pods/4f93d2fa-4486-4f9c-973e-7e0db7272176/volumes"
Nov 03 01:39:37.366356 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:39:37.362547    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:40:07.380713 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:40:07.380669    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:40:19.003890 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:40:19.003859    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 01:40:19.008738 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:40:19.008715    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 01:40:19.026508 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:40:19.026484    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 01:40:19.030841 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:40:19.030822    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbrpr_6cb79353-f35d-4dde-ad87-f0d03352cc49/ovn-acl-logging/0.log"
Nov 03 01:40:22.616007 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:40:22.615966    5012 scope.go:122] "RemoveContainer" containerID="a08c5e04f8ae8f813e9b2f92daa20bc28565a352bae4494f8b8cde142e7b6d4b"
Nov 03 01:40:22.625941 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:40:22.625906    5012 scope.go:122] "RemoveContainer" containerID="19e2eebc21a73590c19ab05015ebdcc8c038c44b1b814eda85f28657b4bea639"
Nov 03 01:40:37.380886 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:40:37.380797    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:40:38.584101 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:40:38.584055    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hstf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:40:38.584519 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:40:38.584140    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hstf4" podUID="c2788e33-32b9-47f6-8d84-d8663aa07a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:41:07.364549 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:41:07.364505    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:41:10.031243 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:41:10.031198    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drgvb"]
Nov 03 01:41:10.041244 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:41:10.041214    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-drgvb"]
Nov 03 01:41:11.242242 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:41:11.242207    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34f87501-6806-40c6-ba6f-51bc93763219" path="/var/lib/kubelet/pods/34f87501-6806-40c6-ba6f-51bc93763219/volumes"
Nov 03 01:41:19.691935 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:41:19.690044    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-cdvd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:41:19.691935 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:41:19.690108    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-cdvd9" podUID="a125db3f-e711-4e19-ac5d-cbd3827be603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:41:19.811933 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:41:19.811252    5012 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hn6jd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:41:19.811933 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:41:19.811324    5012 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hn6jd" podUID="99643392-d8d1-47c9-b9d4-e87e2c837f68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:41:26.024933 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:41:26.024880    5012 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k5stq"]
Nov 03 01:41:26.034962 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:41:26.034934    5012 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k5stq"]
Nov 03 01:41:27.242349 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:41:27.242312    5012 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23689016-fe12-41e3-8740-efcce91800b1" path="/var/lib/kubelet/pods/23689016-fe12-41e3-8740-efcce91800b1/volumes"
Nov 03 01:41:37.362476 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:41:37.362440    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:41:49.292744 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:41:49.292707    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f5fd7bcdf-cdr4b_3be4bd66-b870-41a8-931d-94f258522d39/authorino/0.log"
Nov 03 01:42:02.273423 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:02.273337    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-x5z9p_8c5d5029-6d11-48ee-8771-d2bb8cdcde13/manager/0.log"
Nov 03 01:42:03.624340 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:03.624307    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6cdc687864-vbj7c_a13f8b66-2eca-4a75-a954-5ea44360980d/maas-api/0.log"
Nov 03 01:42:07.363264 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:42:07.363227    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:42:09.000880 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:09.000796    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f5fd7bcdf-cdr4b_3be4bd66-b870-41a8-931d-94f258522d39/authorino/0.log"
Nov 03 01:42:09.242819 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:09.242790    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-kbvsh_087f9032-547d-4eec-b6fe-b6f692ef2b47/manager/0.log"
Nov 03 01:42:09.448834 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:09.448796    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-wz9nt_8a8544a5-55b2-4894-a924-6c408585fe18/manager/1.log"
Nov 03 01:42:10.322840 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:10.322811    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-6mmrg_56d60aa9-02ef-45c9-be0a-37f090753315/manager/0.log"
Nov 03 01:42:10.912839 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:10.912807    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-wz9nt_8a8544a5-55b2-4894-a924-6c408585fe18/manager/0.log"
Nov 03 01:42:11.460562 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:11.460532    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-549f9d654fqchwv_6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e/istio-proxy/0.log"
Nov 03 01:42:11.658525 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:11.658497    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-wrm6m_d989a9a9-bedb-45a3-98a7-ac5ba6929b9d/discovery/0.log"
Nov 03 01:42:11.870915 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:11.870890    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66c6b6454f-8nkg7_851499eb-bc55-426a-be44-0538b929f027/kube-auth-proxy/0.log"
Nov 03 01:42:12.463371 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:12.463343    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-cdvd9_a125db3f-e711-4e19-ac5d-cbd3827be603/istio-proxy/0.log"
Nov 03 01:42:13.067945 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:13.067904    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-hn6jd_99643392-d8d1-47c9-b9d4-e87e2c837f68/istio-proxy/0.log"
Nov 03 01:42:13.263151 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:13.263078    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-hstf4_c2788e33-32b9-47f6-8d84-d8663aa07a99/istio-proxy/0.log"
Nov 03 01:42:14.256774 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:14.256740    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6_3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc/istio-proxy/0.log"
Nov 03 01:42:14.855702 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:14.855671    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7df847f57-khf2f_12eb002e-9765-4443-b488-6cec02b9c961/router/0.log"
Nov 03 01:42:16.006692 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:16.006662    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-q2wkw_beb27702-027a-4c8a-9789-2a59dcf2fb56/main/0.log"
Nov 03 01:42:16.382447 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:16.382416    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-p94dw_5b908c0c-a41f-474b-ae30-5a3478684a98/main/0.log"
Nov 03 01:42:16.583160 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:16.583135    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8prfj7_61a532fa-57a2-486e-a29b-474f5f77cce2/main/0.log"
Nov 03 01:42:17.147651 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:17.147616    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-6nz6h_21760fa1-e581-4657-b63e-d9dd3dde6084/main/0.log"
Nov 03 01:42:22.661600 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:22.661566    5012 scope.go:122] "RemoveContainer" containerID="ccff65683816d9fa2fd5f257c8f108b4ac69d33fc6b488ccb6a33096378fe7b3"
Nov 03 01:42:22.670957 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:22.670935    5012 scope.go:122] "RemoveContainer" containerID="3b7dda6a2ee400d1ac309c00f77e3bd827df815141b5d309d6c506c8e786cb23"
Nov 03 01:42:33.282525 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:33.282492    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bgpx8_6aecdeeb-4efb-413e-a26c-9270468062a8/global-pull-secret-syncer/0.log"
Nov 03 01:42:33.633301 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:33.633268    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-2mp9l_ee7974f8-92b4-498a-bf54-2c3a353f4b49/connection-checker/0.log"
Nov 03 01:42:33.759590 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:33.759561    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-lnk6s_d8606abd-d458-4bee-b150-461becfbb83b/connection-checker/0.log"
Nov 03 01:42:33.885946 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:33.885869    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-lt4f5_913d1e5a-3dfc-4282-b5f0-95c4484b31da/connection-checker/0.log"
Nov 03 01:42:34.376173 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:34.376143    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ld685_47258bdf-947a-4a28-86f4-02871f64941a/konnectivity-agent/0.log"
Nov 03 01:42:34.491145 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:34.491109    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-161.ec2.internal_b4a90a0b8aa58f888d21195c4bcfcc5c/haproxy/0.log"
Nov 03 01:42:37.363012 ip-10-0-133-161 kubenswrapper[5012]: E1103 01:42:37.362977    5012 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xwdjk" containerName="extractor"
Nov 03 01:42:41.235550 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:41.235517    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f5fd7bcdf-cdr4b_3be4bd66-b870-41a8-931d-94f258522d39/authorino/0.log"
Nov 03 01:42:41.855412 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:41.855377    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-kbvsh_087f9032-547d-4eec-b6fe-b6f692ef2b47/manager/0.log"
Nov 03 01:42:41.985251 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:41.985224    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-wz9nt_8a8544a5-55b2-4894-a924-6c408585fe18/manager/1.log"
Nov 03 01:42:41.985405 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:41.985259    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-wz9nt_8a8544a5-55b2-4894-a924-6c408585fe18/manager/0.log"
Nov 03 01:42:42.642579 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:42.642546    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-6mmrg_56d60aa9-02ef-45c9-be0a-37f090753315/manager/0.log"
Nov 03 01:42:44.161820 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.161783    5012 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n428r/perf-node-gather-daemonset-dzvn2"]
Nov 03 01:42:44.162415 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.162387    5012 cpu_manager.go:414] "containerMap: removing container" podUID="db3c4862-4118-42a3-bb11-dbc1d638f12f" containerName="maas-api"
Nov 03 01:42:44.162472 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.162416    5012 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db3c4862-4118-42a3-bb11-dbc1d638f12f" containerName="maas-api"
Nov 03 01:42:44.162525 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.162515    5012 memory_manager.go:365] "RemoveStaleState removing state" podUID="db3c4862-4118-42a3-bb11-dbc1d638f12f" containerName="maas-api"
Nov 03 01:42:44.165704 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.165686    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n428r/perf-node-gather-daemonset-dzvn2"
Nov 03 01:42:44.180599 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.180577    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n428r\"/\"kube-root-ca.crt\""
Nov 03 01:42:44.191145 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.191106    5012 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n428r\"/\"default-dockercfg-dmzvq\""
Nov 03 01:42:44.191145 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.191113    5012 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n428r\"/\"openshift-service-ca.crt\""
Nov 03 01:42:44.195811 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.195787    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n428r/perf-node-gather-daemonset-dzvn2"]
Nov 03 01:42:44.315240 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.315217    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3d88ae2a-8862-4d37-b07d-957c8c8869b0-podres\") pod \"perf-node-gather-daemonset-dzvn2\" (UID: \"3d88ae2a-8862-4d37-b07d-957c8c8869b0\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-dzvn2"
Nov 03 01:42:44.315373 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.315251    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d88ae2a-8862-4d37-b07d-957c8c8869b0-lib-modules\") pod \"perf-node-gather-daemonset-dzvn2\" (UID: \"3d88ae2a-8862-4d37-b07d-957c8c8869b0\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-dzvn2"
Nov 03 01:42:44.315373 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.315267    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmtz4\" (UniqueName: \"kubernetes.io/projected/3d88ae2a-8862-4d37-b07d-957c8c8869b0-kube-api-access-qmtz4\") pod \"perf-node-gather-daemonset-dzvn2\" (UID: \"3d88ae2a-8862-4d37-b07d-957c8c8869b0\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-dzvn2"
Nov 03 01:42:44.315542 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.315522    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3d88ae2a-8862-4d37-b07d-957c8c8869b0-proc\") pod \"perf-node-gather-daemonset-dzvn2\" (UID: \"3d88ae2a-8862-4d37-b07d-957c8c8869b0\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-dzvn2"
Nov 03 01:42:44.315601 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.315552    5012 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d88ae2a-8862-4d37-b07d-957c8c8869b0-sys\") pod \"perf-node-gather-daemonset-dzvn2\" (UID: \"3d88ae2a-8862-4d37-b07d-957c8c8869b0\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-dzvn2"
Nov 03 01:42:44.416543 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.416465    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d88ae2a-8862-4d37-b07d-957c8c8869b0-lib-modules\") pod \"perf-node-gather-daemonset-dzvn2\" (UID: \"3d88ae2a-8862-4d37-b07d-957c8c8869b0\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-dzvn2"
Nov 03 01:42:44.416543 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.416507    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmtz4\" (UniqueName: \"kubernetes.io/projected/3d88ae2a-8862-4d37-b07d-957c8c8869b0-kube-api-access-qmtz4\") pod \"perf-node-gather-daemonset-dzvn2\" (UID: \"3d88ae2a-8862-4d37-b07d-957c8c8869b0\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-dzvn2"
Nov 03 01:42:44.416736 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.416590    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3d88ae2a-8862-4d37-b07d-957c8c8869b0-proc\") pod \"perf-node-gather-daemonset-dzvn2\" (UID: \"3d88ae2a-8862-4d37-b07d-957c8c8869b0\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-dzvn2"
Nov 03 01:42:44.416736 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.416602    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d88ae2a-8862-4d37-b07d-957c8c8869b0-lib-modules\") pod \"perf-node-gather-daemonset-dzvn2\" (UID: \"3d88ae2a-8862-4d37-b07d-957c8c8869b0\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-dzvn2"
Nov 03 01:42:44.416736 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.416619    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d88ae2a-8862-4d37-b07d-957c8c8869b0-sys\") pod \"perf-node-gather-daemonset-dzvn2\" (UID: \"3d88ae2a-8862-4d37-b07d-957c8c8869b0\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-dzvn2"
Nov 03 01:42:44.416736 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.416670    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d88ae2a-8862-4d37-b07d-957c8c8869b0-sys\") pod \"perf-node-gather-daemonset-dzvn2\" (UID: \"3d88ae2a-8862-4d37-b07d-957c8c8869b0\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-dzvn2"
Nov 03 01:42:44.416736 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.416681    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3d88ae2a-8862-4d37-b07d-957c8c8869b0-proc\") pod \"perf-node-gather-daemonset-dzvn2\" (UID: \"3d88ae2a-8862-4d37-b07d-957c8c8869b0\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-dzvn2"
Nov 03 01:42:44.416736 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.416720    5012 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3d88ae2a-8862-4d37-b07d-957c8c8869b0-podres\") pod \"perf-node-gather-daemonset-dzvn2\" (UID: \"3d88ae2a-8862-4d37-b07d-957c8c8869b0\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-dzvn2"
Nov 03 01:42:44.416985 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.416754    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3d88ae2a-8862-4d37-b07d-957c8c8869b0-podres\") pod \"perf-node-gather-daemonset-dzvn2\" (UID: \"3d88ae2a-8862-4d37-b07d-957c8c8869b0\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-dzvn2"
Nov 03 01:42:44.433524 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.433503    5012 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmtz4\" (UniqueName: \"kubernetes.io/projected/3d88ae2a-8862-4d37-b07d-957c8c8869b0-kube-api-access-qmtz4\") pod \"perf-node-gather-daemonset-dzvn2\" (UID: \"3d88ae2a-8862-4d37-b07d-957c8c8869b0\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-dzvn2"
Nov 03 01:42:44.478626 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.478608    5012 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n428r/perf-node-gather-daemonset-dzvn2"
Nov 03 01:42:44.771628 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:44.771596    5012 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n428r/perf-node-gather-daemonset-dzvn2"]
Nov 03 01:42:44.773093 ip-10-0-133-161 kubenswrapper[5012]: W1103 01:42:44.773054    5012 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3d88ae2a_8862_4d37_b07d_957c8c8869b0.slice/crio-3e82cd631a10ee8a86149bf575b6c632b37db6c2686badd98f488bb3f874c2bd WatchSource:0}: Error finding container 3e82cd631a10ee8a86149bf575b6c632b37db6c2686badd98f488bb3f874c2bd: Status 404 returned error can't find the container with id 3e82cd631a10ee8a86149bf575b6c632b37db6c2686badd98f488bb3f874c2bd
Nov 03 01:42:45.073041 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:45.072933    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n428r/perf-node-gather-daemonset-dzvn2" event={"ID":"3d88ae2a-8862-4d37-b07d-957c8c8869b0","Type":"ContainerStarted","Data":"7c8ba80ca425c9b4324787f13b8ded47b7294b1480565bc1a95adfcce1d799a6"}
Nov 03 01:42:45.073041 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:45.073012    5012 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n428r/perf-node-gather-daemonset-dzvn2" event={"ID":"3d88ae2a-8862-4d37-b07d-957c8c8869b0","Type":"ContainerStarted","Data":"3e82cd631a10ee8a86149bf575b6c632b37db6c2686badd98f488bb3f874c2bd"}
Nov 03 01:42:45.073241 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:45.073179    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-n428r/perf-node-gather-daemonset-dzvn2"
Nov 03 01:42:45.108747 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:45.108702    5012 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-n428r/perf-node-gather-daemonset-dzvn2" podStartSLOduration=1.108690204 podStartE2EDuration="1.108690204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:42:45.107179094 +0000 UTC m=+5847.986763138" watchObservedRunningTime="2026-11-03 01:42:45.108690204 +0000 UTC m=+5847.988274247"
Nov 03 01:42:45.122069 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:45.122045    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-ffkwv_dbea6432-5ba3-49f3-8a83-db2605a3aa2c/cluster-monitoring-operator/0.log"
Nov 03 01:42:45.451774 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:45.451745    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-x5z9p_8c5d5029-6d11-48ee-8771-d2bb8cdcde13/manager/0.log"
Nov 03 01:42:46.651838 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:46.651805    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nlbnj_0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5/node-exporter/0.log"
Nov 03 01:42:46.777565 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:46.777535    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nlbnj_0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5/kube-rbac-proxy/0.log"
Nov 03 01:42:46.903890 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:46.903823    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nlbnj_0ad0a0bf-abdc-43d2-b3cb-35f9b7fee8e5/init-textfile/0.log"
Nov 03 01:42:48.816800 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:48.816767    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-x5z9p_8c5d5029-6d11-48ee-8771-d2bb8cdcde13/manager/0.log"
Nov 03 01:42:51.089180 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:51.089151    5012 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-n428r/perf-node-gather-daemonset-dzvn2"
Nov 03 01:42:51.327834 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:51.327804    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-2m6bp_bb5193c7-10f4-4436-9c69-20e6087439c2/networking-console-plugin/0.log"
Nov 03 01:42:52.049579 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:52.049552    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-x5z9p_8c5d5029-6d11-48ee-8771-d2bb8cdcde13/manager/0.log"
Nov 03 01:42:52.504884 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:52.504846    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/2.log"
Nov 03 01:42:52.509192 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:52.509148    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5d95d_c1de1047-c4c9-4d0b-8f72-bafc79987c2e/console-operator/3.log"
Nov 03 01:42:53.993602 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:53.993573    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-549f9d654fqchwv_6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e/istio-proxy/0.log"
Nov 03 01:42:54.110128 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:54.110094    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-wrm6m_d989a9a9-bedb-45a3-98a7-ac5ba6929b9d/discovery/0.log"
Nov 03 01:42:54.167541 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:54.167515    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66c6b6454f-8nkg7_851499eb-bc55-426a-be44-0538b929f027/kube-auth-proxy/0.log"
Nov 03 01:42:54.400830 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:54.400797    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-cdvd9_a125db3f-e711-4e19-ac5d-cbd3827be603/istio-proxy/0.log"
Nov 03 01:42:54.625378 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:54.625343    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-hn6jd_99643392-d8d1-47c9-b9d4-e87e2c837f68/istio-proxy/0.log"
Nov 03 01:42:54.697902 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:54.697838    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-hstf4_c2788e33-32b9-47f6-8d84-d8663aa07a99/istio-proxy/0.log"
Nov 03 01:42:55.018934 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:55.018850    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-v4sr6_eddf03ef-53ea-4fc0-9f3f-da7d741cf06a/volume-data-source-validator/0.log"
Nov 03 01:42:55.032102 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:55.032078    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6_3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc/istio-proxy/0.log"
Nov 03 01:42:55.239984 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:55.239937    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7df847f57-khf2f_12eb002e-9765-4443-b488-6cec02b9c961/router/0.log"
Nov 03 01:42:57.529844 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:57.529812    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lcdlg_7cdb4c55-92a6-4791-8ab8-fcb22ef3a145/dns/0.log"
Nov 03 01:42:57.654273 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:57.654250    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lcdlg_7cdb4c55-92a6-4791-8ab8-fcb22ef3a145/kube-rbac-proxy/0.log"
Nov 03 01:42:58.195514 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:58.195487    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-549f9d654fqchwv_6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e/istio-proxy/0.log"
Nov 03 01:42:58.312505 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:58.312477    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-wrm6m_d989a9a9-bedb-45a3-98a7-ac5ba6929b9d/discovery/0.log"
Nov 03 01:42:58.347923 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:58.347892    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66c6b6454f-8nkg7_851499eb-bc55-426a-be44-0538b929f027/kube-auth-proxy/0.log"
Nov 03 01:42:58.541905 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:58.541831    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gtngg_28e0cef7-ba0d-4546-a193-2f411c441350/dns-node-resolver/0.log"
Nov 03 01:42:58.549051 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:58.549029    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-cdvd9_a125db3f-e711-4e19-ac5d-cbd3827be603/istio-proxy/0.log"
Nov 03 01:42:58.753352 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:58.753326    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-hn6jd_99643392-d8d1-47c9-b9d4-e87e2c837f68/istio-proxy/0.log"
Nov 03 01:42:58.823063 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:58.822994    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-hstf4_c2788e33-32b9-47f6-8d84-d8663aa07a99/istio-proxy/0.log"
Nov 03 01:42:59.159948 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:59.159894    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-95cb67c75-x97b6_3b9b7722-1b39-4fd2-ba12-aa6668d1c8dc/istio-proxy/0.log"
Nov 03 01:42:59.367742 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:59.367712    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7df847f57-khf2f_12eb002e-9765-4443-b488-6cec02b9c961/router/0.log"
Nov 03 01:42:59.696128 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:59.696097    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29894400-s5xp5_daa3da9a-3a5f-49a4-8a09-0036fdc5aa44/image-pruner/0.log"
Nov 03 01:42:59.988401 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:42:59.988322    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f5fd7bcdf-cdr4b_3be4bd66-b870-41a8-931d-94f258522d39/authorino/0.log"
Nov 03 01:43:00.026086 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:43:00.026060    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-kbvsh_087f9032-547d-4eec-b6fe-b6f692ef2b47/manager/0.log"
Nov 03 01:43:00.087519 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:43:00.087495    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-wz9nt_8a8544a5-55b2-4894-a924-6c408585fe18/manager/1.log"
Nov 03 01:43:00.087692 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:43:00.087567    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-wz9nt_8a8544a5-55b2-4894-a924-6c408585fe18/manager/0.log"
Nov 03 01:43:00.100112 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:43:00.100091    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rtkhb_ffb2a03c-414c-4d68-ab86-212c554270e2/node-ca/0.log"
Nov 03 01:43:00.412120 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:43:00.412082    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-6mmrg_56d60aa9-02ef-45c9-be0a-37f090753315/manager/0.log"
Nov 03 01:43:02.445157 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:43:02.445126    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-549f9d654fqchwv_6f019f6c-5c7e-4a62-bf81-b19ea85a6d8e/istio-proxy/0.log"
Nov 03 01:43:02.620785 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:43:02.620758    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-wrm6m_d989a9a9-bedb-45a3-98a7-ac5ba6929b9d/discovery/0.log"
Nov 03 01:43:03.125030 ip-10-0-133-161 kubenswrapper[5012]: I1103 01:43:03.124995    5012 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66c6b6454f-8nkg7_851499eb-bc55-426a-be44-0538b929f027/kube-auth-proxy/0.log"